README

eucrim website

made with wagtail badge

Requirements

libpoppler-cpp-dev
fonts-roboto
fonts-cmu

Setting up database

apt install postgresql postgresql-contrib
sudo su - postgres
postgres@host:$ createuser u_eucrim
postgres@host:$ createdb  eucrim_wagtail --owner u_eucrim
postgres@host:$ psql -c "ALTER USER u_eucrim WITH PASSWORD 'mysecret'"
# pruning and recreate the db:
postres@host:$ dropdb eucrim_wagtail && createdb eucrim_wagtail --owner u_eucrim
# ToDo: dump and restore db:
postgres@host:$ pg_dump eucrim_wagtail > eucrim.sqldump
postgres@host:$ psql eucrim_wagtail < eucrim.sqldump
# Alternative restore method:
# postgres@host:$ pg_restore --schema=public --create --dbname=eucrim_wagtail <path_to_pg_file>  # may throw harmless errors

Hints:

Add DB host and username in our podman-based setup: -h 127.0.0.1 -U u_eucrim

Setting up huey, systemd and memcached

apt install memcached
ln -s /etc/systemd/system/eucrim-huey-task-queue.service  ./etc/systemd/system/eucrim-huey-task-queue.service

>>> from eucrim.procedure.tasks import get_eur_lex_links
>>> res = get_eur_lex_links()

Setting up wagtail project

apt-get install libpython3-dev zlib1g-dev libjpeg-dev libmagickwand-dev pandoc poppler-utils libpoppler-cpp-dev python3-tk # make ruby-sass ruby-listen
git clone git@gitlab.com:tombreit/eucrim.git
cd eucrim/
pyvenv venv
source venv/bin/activate
./manage.py migrate
./manage.py load_initial_data # optional, prepopulate with some data
./manage.py createsuperuser # not needed when load_initial_data was used
./manage.py update_index  # build the initial search index
./manage.py runserver

The wagtail admin interface is now accessible via URL:

http://127.0.0.1:8000/admin/

Fixtures

./manage.py dumpdata --indent 2 \
   -e sessions -e contenttypes \
   -e wagtailsearch -e wagtailimages.rendition \
   -e wagtailcore.pagelogentry -e wagtailcore.pagerevision \
   -e auth.permission -e wagtailcore.groupcollectionpermission -e wagtailcore.grouppagepermission \
   -e wagtailimages.rendition \
   --traceback --natural-primary --natural-foreign  \
   -o tmp/fixture.json

# ./manage.py migrate
# ./manage.py flush  # to get rid of the wagtail initial homepage

./manage.py loaddata tmp/fixture.json

Generate requirements files

pip-compile --output-file=requirements.txt pyproject.toml
pip-compile --output-file=requirements-dev.txt --extra=dev pyproject.toml

Generate documentation

cd docs
make clean
sphinx-apidoc --force --output-dir=source/apidoc/ ../ ../eucrim/*/migrations/*  # regenerate apidocs
make html

Running tests

# install dev requirements (once)
pip install -r requirements-dev.txt

# run the full Django test suite
./manage.py test

# or run tests for a specific app (example)
./manage.py test eucrim.search

Compiling static assetts

# make copy
# make watch
# We're using npm scripts now. For available npm scripts run:
npm run

Refresh fixtures:

# ./manage.py dumpdata --natural-foreign --natural-primary --exclude=sessions.Session --indent 4 article  issue author auth association treaty news core wagtailcore.page wagtailcore.pagerevision contenttypes wagtailimages wagtailcore.collection> /tmp/asso.json
# ./manage.py dumpdata --natural-foreign --natural-primary --exclude contenttypes --exclude auth.Permission --exclude wagtailcore.groupcollectionpermission  --exclude wagtailimages.rendition  --exclude sessions.session --exclude wagtailcore.grouppagepermission --exclude wagtailcore.pagerevision --indent 4 > core/fixtures/eucrim.json
# ./manage.py dumpdata --natural-primary --natural-foreign --indent=2 --exclude=sessions.Session > /tmp/dump.json
# for cat in categoryareascrime categorycooperation categoryfoundation categoryinstitution categoryproceduralcriminallaw categoryregion; do ./manage.py dumpdata core."$cat" --natural-foreign  -e contenttypes --indent 4 >>  /tmp/categories.json; done
# used for creating the eucrim_initial_data.json fixture:
./manage.py dumpdata --natural-primary --natural-foreign --indent=2 --exclude=sessions.Session --exclude=postgres_search.indexentry > eucrim/core/fixtures/eucrim_initial_data.json

Fixtures: Model Associaion

./manage.py dumpdata association --natural-foreign --natural-primary --exclude contenttypes --exclude auth.Permission --exclude wagtailcore.groupcollectionpermission  --exclude wagtailimages.rendition  --exclude sessions.session --exclude wagtailcore.grouppagepermission --exclude wagtailcore.pagerevision --indent 4 > core/fixtures/associations.json

Importing issue pdfs

Importing legacy pdfs via a custom django management command import_pdfs.

./manage.py import_pdfs -h
./manage.py import_pdfs path-to-dir-containing-pdfs

Resetting migrations

Reference: https://simpleisbetterthancomplex.com/tutorial/2016/07/26/how-to-reset-migrations.html

find . -not -path '*/\.*' -path "*/migrations/*.py" -not -name "__init__.py" -delete
find . -path "*/migrations/*.pyc"  -delete
python manage.py makemigrations

LLMs.txt Integration

eucrim exposes content for LLMs following the llms.txt standard:

GET /llms.txt — Site-level content index listing all articles, news, and author profiles with brief descriptions
GET /llms-full.txt — Full concatenated content dump (all articles, news, and profiles in one file)
GET /<page-url>.md — Per-page Markdown export for any article, news item, or author profile

For example:

curl https://eucrim.eu/llms.txt
curl https://eucrim.eu/articles/my-article.md
curl https://eucrim.eu/news/my-news.md
curl https://eucrim.eu/authors/my-author.md
curl https://eucrim.eu/llms-full.txt

Hints

django-select2 not working? Check if a “real” cache (like memcached) is configured and running.