3
0
Fork 0

[PRT] web_no_crawler/12.0

12.0
Stefan Rijnhart 2020-06-25 17:30:45 +02:00
parent 03acd2bb49
commit 8697d5fc48
7 changed files with 32 additions and 551 deletions

View File

@ -1,101 +1,35 @@
===========================================
Disallow indexing completely via robots.txt
===========================================
**This file is going to be generated by oca-gen-addon-readme.**
.. !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! This file is generated by oca-gen-addon-readme !!
!! changes will be overwritten. !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
*Manual changes will be overwritten.*
.. |badge1| image:: https://img.shields.io/badge/maturity-Beta-yellow.png
:target: https://odoo-community.org/page/development-status
:alt: Beta
.. |badge2| image:: https://img.shields.io/badge/licence-AGPL--3-blue.png
:target: http://www.gnu.org/licenses/agpl-3.0-standalone.html
:alt: License: AGPL-3
.. |badge3| image:: https://img.shields.io/badge/github-OCA%2Fweb-lightgray.png?logo=github
:target: https://github.com/OCA/web/tree/8.0/web_no_crawler
:alt: OCA/web
.. |badge4| image:: https://img.shields.io/badge/weblate-Translate%20me-F47D42.png
:target: https://translation.odoo-community.org/projects/web-8-0/web-8-0-web_no_crawler
:alt: Translate me on Weblate
.. |badge5| image:: https://img.shields.io/badge/runbot-Try%20me-875A7B.png
:target: https://runbot.odoo-community.org/runbot/162/8.0
:alt: Try me on Runbot
Please provide content in the ``readme`` directory:
|badge1| |badge2| |badge3| |badge4| |badge5|
* **DESCRIPTION.rst** (required)
* INSTALL.rst (optional)
* CONFIGURE.rst (optional)
* **USAGE.rst** (optional, highly recommended)
* DEVELOP.rst (optional)
* ROADMAP.rst (optional)
* HISTORY.rst (optional, recommended)
* **CONTRIBUTORS.rst** (optional, highly recommended)
* CREDITS.rst (optional)
This module was written to implement a robots.txt file to prevent web crawlers (like google) from indexing pages.
This module does not depend on Website module.
Content of this README will also be drawn from the addon manifest,
from keys such as name, authors, maintainers, development_status,
and license.
**Table of contents**
A good, one sentence summary in the manifest is also highly recommended.
.. contents::
:local:
Installation
============
Automatic changelog generation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Installation as usual. No specific installation steps / configuration required.
`HISTORY.rst` can be auto generated using `towncrier <https://pypi.org/project/towncrier>`_.
**WARNING:** this module is not to be used with `Website` module as it has a separate functionality for robots.txt.
Just put towncrier compatible changelog fragments into `readme/newsfragments`
and the changelog file will be automatically generated and updated when a new fragment is added.
Usage
=====
Please refer to `towncrier` documentation to know more.
To use this module, you need to:
No configuration needed. Once installed adds robots.txt (ex.: http://example.org/robots.txt).
Known issues / Roadmap
======================
* Investigate possibilies for compatibility with `Website` module as it has a separate functionality for robots.txt.
Bug Tracker
===========
Bugs are tracked on `GitHub Issues <https://github.com/OCA/web/issues>`_.
In case of trouble, please check there if your issue has already been reported.
If you spotted it first, help us smashing it by providing a detailed and welcomed
`feedback <https://github.com/OCA/web/issues/new?body=module:%20web_no_crawler%0Aversion:%208.0%0A%0A**Steps%20to%20reproduce**%0A-%20...%0A%0A**Current%20behavior**%0A%0A**Expected%20behavior**>`_.
Do not contact contributors directly about support or help with technical issues.
Credits
=======
Authors
~~~~~~~
* Ventor
* Xpansa Group
Contributors
~~~~~~~~~~~~
* Nedas Zilinskas <nedas.zilinskas@xpansa.com> (http://xpansa.com)
Other credits
~~~~~~~~~~~~~
The development of this module has been financially supported by:
* Ventor, Xpansa Group (<https://ventor.tech/>)
Maintainers
~~~~~~~~~~~
This module is maintained by the OCA.
.. image:: https://odoo-community.org/logo.png
:alt: Odoo Community Association
:target: https://odoo-community.org
OCA, or the Odoo Community Association, is a nonprofit organization whose
mission is to support the collaborative development of Odoo features and
promote its widespread use.
This module is part of the `OCA/web <https://github.com/OCA/web/tree/8.0/web_no_crawler>`_ project on GitHub.
You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute.
NOTE: the changelog will be automatically generated when using `/ocabot merge $option`.
If you need to run it manually, refer to `OCA/maintainer-tools README <https://github.com/OCA/maintainer-tools>`_.

View File

@ -1,4 +1,3 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2018 Ventor, Xpansa Group (<https://ventor.tech/>)
# License LGPL-3.0 or later (http://www.gnu.org/licenses/lgpl.html).

View File

@ -1,17 +1,15 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2018 Ventor, Xpansa Group (<https://ventor.tech/>)
# License LGPL-3.0 or later (http://www.gnu.org/licenses/lgpl.html).
{
'name': 'Disallow indexing completely via robots.txt',
'author': 'Ventor, Xpansa Group, Odoo Community Association (OCA)',
'website': 'https://github.com/OCA/website/tree/8.0',
'website': 'https://github.com/OCA/web',
'category': 'web',
'version': '8.0.1.0.0',
'version': '12.0.1.0.0',
'depends': [
'base',
'web',
],
'installable': True,
'application': False,
'license': 'AGPL-3',
'license': 'LGPL-3',
}

View File

@ -1,4 +1,3 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2018 Ventor, Xpansa Group (<https://ventor.tech/>)
# License LGPL-3.0 or later (http://www.gnu.org/licenses/lgpl.html).

View File

@ -1,12 +1,10 @@
# -*- coding: utf-8 -*-
# Copyright (C) 2018 Ventor, Xpansa Group (<https://ventor.tech/>)
# License LGPL-3.0 or later (http://www.gnu.org/licenses/lgpl.html).
import openerp
from openerp import http
from odoo import http
from odoo.addons.web.controllers.main import Home
class Main(openerp.addons.web.controllers.main.Home):
class Main(Home):
@http.route('/robots.txt', type='http', auth="none")
def robots(self):

View File

@ -1 +1,2 @@
* Nedas Zilinskas <nedas.zilinskas@xpansa.com> (http://xpansa.com)
* Stefan Rijnhart <stefan@opener.amsterdam>

View File

@ -1,448 +0,0 @@
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.15.1: http://docutils.sourceforge.net/" />
<title>Disallow indexing completely via robots.txt</title>
<style type="text/css">
/*
:Author: David Goodger (goodger@python.org)
:Id: $Id: html4css1.css 7952 2016-07-26 18:15:59Z milde $
:Copyright: This stylesheet has been placed in the public domain.
Default cascading style sheet for the HTML output of Docutils.
See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to
customize this style sheet.
*/
/* used to remove borders from tables and images */
.borderless, table.borderless td, table.borderless th {
border: 0 }
table.borderless td, table.borderless th {
/* Override padding for "table.docutils td" with "! important".
The right padding separates the table cells. */
padding: 0 0.5em 0 0 ! important }
.first {
/* Override more specific margin styles with "! important". */
margin-top: 0 ! important }
.last, .with-subtitle {
margin-bottom: 0 ! important }
.hidden {
display: none }
.subscript {
vertical-align: sub;
font-size: smaller }
.superscript {
vertical-align: super;
font-size: smaller }
a.toc-backref {
text-decoration: none ;
color: black }
blockquote.epigraph {
margin: 2em 5em ; }
dl.docutils dd {
margin-bottom: 0.5em }
object[type="image/svg+xml"], object[type="application/x-shockwave-flash"] {
overflow: hidden;
}
/* Uncomment (and remove this text!) to get bold-faced definition list terms
dl.docutils dt {
font-weight: bold }
*/
div.abstract {
margin: 2em 5em }
div.abstract p.topic-title {
font-weight: bold ;
text-align: center }
div.admonition, div.attention, div.caution, div.danger, div.error,
div.hint, div.important, div.note, div.tip, div.warning {
margin: 2em ;
border: medium outset ;
padding: 1em }
div.admonition p.admonition-title, div.hint p.admonition-title,
div.important p.admonition-title, div.note p.admonition-title,
div.tip p.admonition-title {
font-weight: bold ;
font-family: sans-serif }
div.attention p.admonition-title, div.caution p.admonition-title,
div.danger p.admonition-title, div.error p.admonition-title,
div.warning p.admonition-title, .code .error {
color: red ;
font-weight: bold ;
font-family: sans-serif }
/* Uncomment (and remove this text!) to get reduced vertical space in
compound paragraphs.
div.compound .compound-first, div.compound .compound-middle {
margin-bottom: 0.5em }
div.compound .compound-last, div.compound .compound-middle {
margin-top: 0.5em }
*/
div.dedication {
margin: 2em 5em ;
text-align: center ;
font-style: italic }
div.dedication p.topic-title {
font-weight: bold ;
font-style: normal }
div.figure {
margin-left: 2em ;
margin-right: 2em }
div.footer, div.header {
clear: both;
font-size: smaller }
div.line-block {
display: block ;
margin-top: 1em ;
margin-bottom: 1em }
div.line-block div.line-block {
margin-top: 0 ;
margin-bottom: 0 ;
margin-left: 1.5em }
div.sidebar {
margin: 0 0 0.5em 1em ;
border: medium outset ;
padding: 1em ;
background-color: #ffffee ;
width: 40% ;
float: right ;
clear: right }
div.sidebar p.rubric {
font-family: sans-serif ;
font-size: medium }
div.system-messages {
margin: 5em }
div.system-messages h1 {
color: red }
div.system-message {
border: medium outset ;
padding: 1em }
div.system-message p.system-message-title {
color: red ;
font-weight: bold }
div.topic {
margin: 2em }
h1.section-subtitle, h2.section-subtitle, h3.section-subtitle,
h4.section-subtitle, h5.section-subtitle, h6.section-subtitle {
margin-top: 0.4em }
h1.title {
text-align: center }
h2.subtitle {
text-align: center }
hr.docutils {
width: 75% }
img.align-left, .figure.align-left, object.align-left, table.align-left {
clear: left ;
float: left ;
margin-right: 1em }
img.align-right, .figure.align-right, object.align-right, table.align-right {
clear: right ;
float: right ;
margin-left: 1em }
img.align-center, .figure.align-center, object.align-center {
display: block;
margin-left: auto;
margin-right: auto;
}
table.align-center {
margin-left: auto;
margin-right: auto;
}
.align-left {
text-align: left }
.align-center {
clear: both ;
text-align: center }
.align-right {
text-align: right }
/* reset inner alignment in figures */
div.align-right {
text-align: inherit }
/* div.align-center * { */
/* text-align: left } */
.align-top {
vertical-align: top }
.align-middle {
vertical-align: middle }
.align-bottom {
vertical-align: bottom }
ol.simple, ul.simple {
margin-bottom: 1em }
ol.arabic {
list-style: decimal }
ol.loweralpha {
list-style: lower-alpha }
ol.upperalpha {
list-style: upper-alpha }
ol.lowerroman {
list-style: lower-roman }
ol.upperroman {
list-style: upper-roman }
p.attribution {
text-align: right ;
margin-left: 50% }
p.caption {
font-style: italic }
p.credits {
font-style: italic ;
font-size: smaller }
p.label {
white-space: nowrap }
p.rubric {
font-weight: bold ;
font-size: larger ;
color: maroon ;
text-align: center }
p.sidebar-title {
font-family: sans-serif ;
font-weight: bold ;
font-size: larger }
p.sidebar-subtitle {
font-family: sans-serif ;
font-weight: bold }
p.topic-title {
font-weight: bold }
pre.address {
margin-bottom: 0 ;
margin-top: 0 ;
font: inherit }
pre.literal-block, pre.doctest-block, pre.math, pre.code {
margin-left: 2em ;
margin-right: 2em }
pre.code .ln { color: grey; } /* line numbers */
pre.code, code { background-color: #eeeeee }
pre.code .comment, code .comment { color: #5C6576 }
pre.code .keyword, code .keyword { color: #3B0D06; font-weight: bold }
pre.code .literal.string, code .literal.string { color: #0C5404 }
pre.code .name.builtin, code .name.builtin { color: #352B84 }
pre.code .deleted, code .deleted { background-color: #DEB0A1}
pre.code .inserted, code .inserted { background-color: #A3D289}
span.classifier {
font-family: sans-serif ;
font-style: oblique }
span.classifier-delimiter {
font-family: sans-serif ;
font-weight: bold }
span.interpreted {
font-family: sans-serif }
span.option {
white-space: nowrap }
span.pre {
white-space: pre }
span.problematic {
color: red }
span.section-subtitle {
/* font-size relative to parent (h1..h6 element) */
font-size: 80% }
table.citation {
border-left: solid 1px gray;
margin-left: 1px }
table.docinfo {
margin: 2em 4em }
table.docutils {
margin-top: 0.5em ;
margin-bottom: 0.5em }
table.footnote {
border-left: solid 1px black;
margin-left: 1px }
table.docutils td, table.docutils th,
table.docinfo td, table.docinfo th {
padding-left: 0.5em ;
padding-right: 0.5em ;
vertical-align: top }
table.docutils th.field-name, table.docinfo th.docinfo-name {
font-weight: bold ;
text-align: left ;
white-space: nowrap ;
padding-left: 0 }
/* "booktabs" style (no vertical lines) */
table.docutils.booktabs {
border: 0px;
border-top: 2px solid;
border-bottom: 2px solid;
border-collapse: collapse;
}
table.docutils.booktabs * {
border: 0px;
}
table.docutils.booktabs th {
border-bottom: thin solid;
text-align: left;
}
h1 tt.docutils, h2 tt.docutils, h3 tt.docutils,
h4 tt.docutils, h5 tt.docutils, h6 tt.docutils {
font-size: 100% }
ul.auto-toc {
list-style-type: none }
</style>
</head>
<body>
<div class="document" id="disallow-indexing-completely-via-robots-txt">
<h1 class="title">Disallow indexing completely via robots.txt</h1>
<!-- !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! This file is generated by oca-gen-addon-readme !!
!! changes will be overwritten. !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! -->
<p><a class="reference external" href="https://odoo-community.org/page/development-status"><img alt="Beta" src="https://img.shields.io/badge/maturity-Beta-yellow.png" /></a> <a class="reference external" href="http://www.gnu.org/licenses/agpl-3.0-standalone.html"><img alt="License: AGPL-3" src="https://img.shields.io/badge/licence-AGPL--3-blue.png" /></a> <a class="reference external" href="https://github.com/OCA/web/tree/8.0/web_no_crawler"><img alt="OCA/web" src="https://img.shields.io/badge/github-OCA%2Fweb-lightgray.png?logo=github" /></a> <a class="reference external" href="https://translation.odoo-community.org/projects/web-8-0/web-8-0-web_no_crawler"><img alt="Translate me on Weblate" src="https://img.shields.io/badge/weblate-Translate%20me-F47D42.png" /></a> <a class="reference external" href="https://runbot.odoo-community.org/runbot/162/8.0"><img alt="Try me on Runbot" src="https://img.shields.io/badge/runbot-Try%20me-875A7B.png" /></a></p>
<p>This module was written to implement a robots.txt file to prevent web crawlers (like google) from indexing pages.
This module does not depend on Website module.</p>
<p><strong>Table of contents</strong></p>
<div class="contents local topic" id="contents">
<ul class="simple">
<li><a class="reference internal" href="#installation" id="id1">Installation</a></li>
<li><a class="reference internal" href="#usage" id="id2">Usage</a></li>
<li><a class="reference internal" href="#known-issues-roadmap" id="id3">Known issues / Roadmap</a></li>
<li><a class="reference internal" href="#bug-tracker" id="id4">Bug Tracker</a></li>
<li><a class="reference internal" href="#credits" id="id5">Credits</a><ul>
<li><a class="reference internal" href="#authors" id="id6">Authors</a></li>
<li><a class="reference internal" href="#contributors" id="id7">Contributors</a></li>
<li><a class="reference internal" href="#other-credits" id="id8">Other credits</a></li>
<li><a class="reference internal" href="#maintainers" id="id9">Maintainers</a></li>
</ul>
</li>
</ul>
</div>
<div class="section" id="installation">
<h1><a class="toc-backref" href="#id1">Installation</a></h1>
<p>Installation as usual. No specific installation steps / configuration required.</p>
<p><strong>WARNING:</strong> this module is not to be used with <cite>Website</cite> module as it has a separate functionality for robots.txt.</p>
</div>
<div class="section" id="usage">
<h1><a class="toc-backref" href="#id2">Usage</a></h1>
<p>To use this module, you need to:</p>
<p>No configuration needed. Once installed adds robots.txt (ex.: <a class="reference external" href="http://example.org/robots.txt">http://example.org/robots.txt</a>).</p>
</div>
<div class="section" id="known-issues-roadmap">
<h1><a class="toc-backref" href="#id3">Known issues / Roadmap</a></h1>
<ul class="simple">
<li>Investigate possibilies for compatibility with <cite>Website</cite> module as it has a separate functionality for robots.txt.</li>
</ul>
</div>
<div class="section" id="bug-tracker">
<h1><a class="toc-backref" href="#id4">Bug Tracker</a></h1>
<p>Bugs are tracked on <a class="reference external" href="https://github.com/OCA/web/issues">GitHub Issues</a>.
In case of trouble, please check there if your issue has already been reported.
If you spotted it first, help us smashing it by providing a detailed and welcomed
<a class="reference external" href="https://github.com/OCA/web/issues/new?body=module:%20web_no_crawler%0Aversion:%208.0%0A%0A**Steps%20to%20reproduce**%0A-%20...%0A%0A**Current%20behavior**%0A%0A**Expected%20behavior**">feedback</a>.</p>
<p>Do not contact contributors directly about support or help with technical issues.</p>
</div>
<div class="section" id="credits">
<h1><a class="toc-backref" href="#id5">Credits</a></h1>
<div class="section" id="authors">
<h2><a class="toc-backref" href="#id6">Authors</a></h2>
<ul class="simple">
<li>Ventor</li>
<li>Xpansa Group</li>
</ul>
</div>
<div class="section" id="contributors">
<h2><a class="toc-backref" href="#id7">Contributors</a></h2>
<ul class="simple">
<li>Nedas Zilinskas &lt;<a class="reference external" href="mailto:nedas.zilinskas&#64;xpansa.com">nedas.zilinskas&#64;xpansa.com</a>&gt; (<a class="reference external" href="http://xpansa.com">http://xpansa.com</a>)</li>
</ul>
</div>
<div class="section" id="other-credits">
<h2><a class="toc-backref" href="#id8">Other credits</a></h2>
<p>The development of this module has been financially supported by:</p>
<ul class="simple">
<li>Ventor, Xpansa Group (&lt;<a class="reference external" href="https://ventor.tech/">https://ventor.tech/</a>&gt;)</li>
</ul>
</div>
<div class="section" id="maintainers">
<h2><a class="toc-backref" href="#id9">Maintainers</a></h2>
<p>This module is maintained by the OCA.</p>
<a class="reference external image-reference" href="https://odoo-community.org"><img alt="Odoo Community Association" src="https://odoo-community.org/logo.png" /></a>
<p>OCA, or the Odoo Community Association, is a nonprofit organization whose
mission is to support the collaborative development of Odoo features and
promote its widespread use.</p>
<p>This module is part of the <a class="reference external" href="https://github.com/OCA/web/tree/8.0/web_no_crawler">OCA/web</a> project on GitHub.</p>
<p>You are welcome to contribute. To learn how please visit <a class="reference external" href="https://odoo-community.org/page/Contribute">https://odoo-community.org/page/Contribute</a>.</p>
</div>
</div>
</div>
</body>
</html>