Merge branch 'master' into federation_merge_from_master_2

61 files changed, 1175 insertions, 1078 deletions

diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md
index e0270d4c..71d4455d 100644
--- a/CODE_OF_CONDUCT.md
+++ b/CODE_OF_CONDUCT.md
@@ -30,6 +30,6 @@ In the Lemmy community we strive to go the extra step to look out for each other
 
 And if someone takes issue with something you said or did, resist the urge to be defensive. Just stop doing what it was they complained about and apologize. Even if you feel you were misinterpreted or unfairly accused, chances are good there was something you could’ve communicated better — remember that it’s your responsibility to make others comfortable. Everyone wants to get along and we are all here first and foremost because we want to talk about cool technology. You will find that people will be eager to assume good intent and forgive as long as you earn their trust.
 
-The enforcement policies listed above apply to all official Lemmy venues; including git repositories under [github.com/LemmyNet/lemmy](https://github.com/LemmyNet/lemmy) and [yerbamate.dev/dessalines/lemmy](https://yerbamate.dev/dessalines/lemmy), the [Matrix channel](https://matrix.to/#/!BZVTUuEiNmRcbFeLeI:matrix.org?via=matrix.org&via=privacytools.io&via=permaweb.io); and all instances under lemmy.ml. For other projects adopting the Rust Code of Conduct, please contact the maintainers of those projects for enforcement. If you wish to use this code of conduct for your own project, consider explicitly mentioning your moderation policy or making a copy with your own moderation policy so as to avoid confusion.
+The enforcement policies listed above apply to all official Lemmy venues; including git repositories under [github.com/LemmyNet/lemmy](https://github.com/LemmyNet/lemmy) and [yerbamate.dev/LemmyNet/lemmy](https://yerbamate.dev/LemmyNet/lemmy), the [Matrix channel](https://matrix.to/#/!BZVTUuEiNmRcbFeLeI:matrix.org?via=matrix.org&via=privacytools.io&via=permaweb.io); and all instances under lemmy.ml. For other projects adopting the Rust Code of Conduct, please contact the maintainers of those projects for enforcement. If you wish to use this code of conduct for your own project, consider explicitly mentioning your moderation policy or making a copy with your own moderation policy so as to avoid confusion.
 
 Adapted from the [Rust Code of Conduct](https://www.rust-lang.org/policies/code-of-conduct), which is based on the [Node.js Policy on Trolling](http://blog.izs.me/post/30036893703/policy-on-trolling) as well as the [Contributor Covenant v1.3.0](https://www.contributor-covenant.org/version/1/3/0/).
diff --git a/RELEASES.md b/RELEASES.md
index 5a4c7645..9946ae32 100644
--- a/RELEASES.md
+++ b/RELEASES.md
@@ -1,3 +1,66 @@
+# Lemmy v0.7.0 Release (2020-06-23)
+
+This release replaces [pictshare](https://github.com/HaschekSolutions/pictshare)
+with [pict-rs](https://git.asonix.dog/asonix/pict-rs), which improves performance
+and security.
+
+Overall, since our last major release in January (v0.6.0), we have closed over
+[100 issues!](https://github.com/LemmyNet/lemmy/milestone/16?closed=1)
+
+- Site-wide list of recent comments
+- Reconnecting websockets
+- Many more themes, including a default light one.
+- Expandable embeds for post links (and thumbnails), from
+[iframely](https://github.com/itteco/iframely)
+- Better icons
+- Emoji autocomplete to post and message bodies, and an Emoji Picker
+- Post body now searchable
+- Community title and description is now searchable
+- Simplified cross-posts
+- Better documentation
+- LOTS more languages
+- Lots of bugs squashed
+- And more ...
+
+## Upgrading
+
+Before starting the upgrade, make sure that you have a working backup of your
+database and image files. See our
+[documentation](https://dev.lemmy.ml/docs/administration_backup_and_restore.html)
+for backup instructions.
+
+**With Ansible:**
+
+```
+# deploy with ansible from your local lemmy git repo
+git pull
+cd ansible
+ansible-playbook lemmy.yml
+# connect via ssh to run the migration script
+ssh your-server
+cd /lemmy/
+wget https://raw.githubusercontent.com/LemmyNet/lemmy/master/docker/prod/migrate-pictshare-to-pictrs.bash
+chmod +x migrate-pictshare-to-pictrs.bash
+sudo ./migrate-pictshare-to-pictrs.bash
+```
+
+**With manual Docker installation:**
+```
+# run these commands on your server
+cd /lemmy
+wget https://raw.githubusercontent.com/LemmyNet/lemmy/master/ansible/templates/nginx.conf
+# Replace the {{ vars }}
+sudo mv nginx.conf /etc/nginx/sites-enabled/lemmy.conf
+sudo nginx -s reload
+wget https://raw.githubusercontent.com/LemmyNet/lemmy/master/docker/prod/docker-compose.yml
+wget https://raw.githubusercontent.com/LemmyNet/lemmy/master/docker/prod/migrate-pictshare-to-pictrs.bash
+chmod +x migrate-pictshare-to-pictrs.bash
+sudo bash migrate-pictshare-to-pictrs.bash
+```
+
+**Note:** After upgrading, all users need to reload the page, then logout and
+login again, so that images are loaded correctly.
+
 # Lemmy v0.6.0 Release (2020-01-16)
 
 `v0.6.0` is here, and we've closed [41 issues!](https://github.com/LemmyNet/lemmy/milestone/15?closed=1) 
diff --git a/ansible/VERSION b/ansible/VERSION
index e31dcbc4..8b20e485 100644
--- a/ansible/VERSION
+++ b/ansible/VERSION
@@ -1 +1 @@
-v0.6.71
+v0.7.0
diff --git a/ansible/ansible.cfg b/ansible/ansible.cfg
index 960a7c40..74b6ab2f 100644
--- a/ansible/ansible.cfg
+++ b/ansible/ansible.cfg
@@ -1,5 +1,6 @@
 [defaults]
 inventory=inventory
+interpreter_python=/usr/bin/python3
 
 [ssh_connection]
 pipelining = True
diff --git a/ansible/lemmy.yml b/ansible/lemmy.yml
index bc01623f..7b78ab8d 100644
--- a/ansible/lemmy.yml
+++ b/ansible/lemmy.yml
@@ -24,10 +24,11 @@
       creates: '/etc/letsencrypt/live/{{domain}}/privkey.pem'
 
   - name: create lemmy folder
-    file: path={{item.path}} state=directory
+    file: path={{item.path}} {{item.owner}} state=directory
     with_items:
-      - { path: '/lemmy/' }
-      - { path: '/lemmy/volumes/' }
+      - { path: '/lemmy/', owner: 'root' }
+      - { path: '/lemmy/volumes/', owner: 'root' }
+      - { path: '/lemmy/volumes/pictrs/', owner: '991' }
 
   - block:
     - name:  add template files
@@ -59,6 +60,7 @@
       project_src: /lemmy/
       state: present
       pull: yes
+      remove_orphans: yes
 
   - name: reload nginx with new config
     shell: nginx -s reload
diff --git a/ansible/lemmy_dev.yml b/ansible/lemmy_dev.yml
index e9b8364f..7a368361 100644
--- a/ansible/lemmy_dev.yml
+++ b/ansible/lemmy_dev.yml
@@ -26,10 +26,11 @@
       creates: '/etc/letsencrypt/live/{{domain}}/privkey.pem'
 
   - name: create lemmy folder
-    file: path={{item.path}} state=directory
+    file: path={{item.path}} owner={{item.owner}} state=directory
     with_items:
-      - { path: '/lemmy/' }
-      - { path: '/lemmy/volumes/' }
+      - { path: '/lemmy/', owner: 'root' }
+      - { path: '/lemmy/volumes/', owner: 'root' }
+      - { path: '/lemmy/volumes/pictrs/', owner: '991' }
 
   - block:
     - name:  add template files
@@ -88,6 +89,7 @@
       project_src: /lemmy/
       state: present
       recreate: always
+      remove_orphans: yes
     ignore_errors: yes
 
   - name: reload nginx with new config
diff --git a/ansible/templates/docker-compose.yml b/ansible/templates/docker-compose.yml
index 9ec1bfbc..f4c94fd7 100644
--- a/ansible/templates/docker-compose.yml
+++ b/ansible/templates/docker-compose.yml
@@ -12,7 +12,7 @@ services:
       - ./lemmy.hjson:/config/config.hjson:ro
     depends_on:
       - postgres
-      - pictshare
+      - pictrs
       - iframely
 
   postgres:
@@ -25,12 +25,13 @@ services:
       - ./volumes/postgres:/var/lib/postgresql/data
     restart: always
 
-  pictshare:
-    image: hascheksolutions/pictshare:latest
+  pictrs:
+    image: asonix/pictrs:amd64-v0.1.0-r9
+    user: 991:991
     ports:
-      - "127.0.0.1:8537:80"
+      - "127.0.0.1:8537:8080"
     volumes:
-      - ./volumes/pictshare:/usr/share/nginx/html/data
+      - ./volumes/pictrs:/mnt
     restart: always
 
   iframely:
diff --git a/ansible/templates/nginx.conf b/ansible/templates/nginx.conf
index a978c189..b710fdb3 100644
--- a/ansible/templates/nginx.conf
+++ b/ansible/templates/nginx.conf
@@ -48,8 +48,8 @@ server {
     add_header X-Frame-Options "DENY";
     add_header X-XSS-Protection "1; mode=block";
 
-    # Upload limit for pictshare
-    client_max_body_size 50M;
+    # Upload limit for pictrs
+    client_max_body_size 20M;
 
     location / {
         proxy_pass http://0.0.0.0:8536;
@@ -70,15 +70,21 @@ server {
         proxy_cache_min_uses    5;
     }    
 
-    location /pictshare/ {
-      proxy_pass http://0.0.0.0:8537/;
-      proxy_set_header X-Real-IP $remote_addr;
-      proxy_set_header Host $host;
-      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+    # Redirect pictshare images to pictrs
+    location ~ /pictshare/(.*)$ {
+      return 301 /pictrs/image/$1;
+    }
 
-      if ($request_uri ~ \.(?:ico|gif|jpe?g|png|webp|bmp|mp4)$) {
-        add_header Cache-Control "public, max-age=31536000, immutable";
-      }   
+    # pict-rs images
+    location /pictrs {
+      location /pictrs/image {
+        proxy_pass http://0.0.0.0:8537/image;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header Host $host;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+      }
+      # Block the import
+      return 403;
     }
 
     location /iframely/ {
diff --git a/docker/dev/Dockerfile b/docker/dev/Dockerfile
index d9ffc2f3..82a03f3c 100644
--- a/docker/dev/Dockerfile
+++ b/docker/dev/Dockerfile
@@ -21,17 +21,13 @@ COPY server/Cargo.toml server/Cargo.lock ./
 RUN sudo chown -R rust:rust .
 RUN mkdir -p ./src/bin \
   && echo 'fn main() { println!("Dummy") }' > ./src/bin/main.rs 
-RUN cargo build --release
+RUN cargo build
 RUN rm -f ./target/x86_64-unknown-linux-musl/release/deps/lemmy_server*
 COPY server/src ./src/
 COPY server/migrations ./migrations/
 
-# Build for release
-RUN cargo build --frozen --release
-
-# Get diesel-cli on there just in case
-# RUN cargo install diesel_cli --no-default-features --features postgres
-
+# Build for debug
+RUN cargo build
 
 FROM ekidd/rust-musl-builder:1.42.0-openssl11 as docs
 WORKDIR /app
@@ -39,15 +35,14 @@ COPY docs ./docs
 RUN sudo chown -R rust:rust .
 RUN mdbook build docs/
 
-
-FROM alpine:3.10
+FROM alpine:3.12
 
 # Install libpq for postgres
 RUN apk add libpq
 
 # Copy resources
 COPY server/config/defaults.hjson /config/defaults.hjson
-COPY --from=rust /app/server/target/x86_64-unknown-linux-musl/release/lemmy_server /app/lemmy
+COPY --from=rust /app/server/target/x86_64-unknown-linux-musl/debug/lemmy_server /app/lemmy
 COPY --from=docs /app/docs/book/ /app/dist/documentation/
 COPY --from=node /app/ui/dist /app/dist
 
diff --git a/docker/dev/Dockerfile.aarch64 b/docker/dev/Dockerfile.aarch64
deleted file mode 100644
index 9636a590..00000000
--- a/docker/dev/Dockerfile.aarch64
+++ /dev/null
@@ -1,79 +0,0 @@
-FROM node:10-jessie as node
-
-WORKDIR /app/ui
-
-# Cache deps
-COPY ui/package.json ui/yarn.lock ./
-RUN yarn install --pure-lockfile
-
-# Build
-COPY ui /app/ui
-RUN yarn build
-
-
-# contains qemu-*-static for cross-compilation
-FROM multiarch/qemu-user-static as qemu
-
-
-FROM arm64v8/rust:1.40-buster as rust
-
-COPY --from=qemu /usr/bin/qemu-aarch64-static /usr/bin
-#COPY --from=qemu /usr/bin/qemu-arm-static /usr/bin
-
-
-# Install musl
-#RUN apt-get update && apt-get install -y mc
-#RUN apt-get install -y musl-tools mc
-#libpq-dev mc
-#RUN rustup target add ${TARGET}
-
-# Cache deps
-WORKDIR /app
-RUN USER=root cargo new server
-WORKDIR /app/server
-COPY server/Cargo.toml server/Cargo.lock ./
-RUN  mkdir -p ./src/bin \
-  && echo 'fn main() { println!("Dummy") }' > ./src/bin/main.rs
-RUN cargo build --release
-# RUN cargo build
-COPY server/src ./src/
-COPY server/migrations ./migrations/
-RUN rm -f ./target/release/deps/lemmy_server* ; rm -f ./target/debug/deps/lemmy_server*
-
-
-# build for release
-RUN cargo build --frozen --release
-# RUN cargo build --frozen
-
-# Get diesel-cli on there just in case
-# RUN cargo install diesel_cli --no-default-features --features postgres
-
-# RUN cp /app/server/target/debug/lemmy_server /app/server/ready
-RUN cp /app/server/target/release/lemmy_server /app/server/ready
-
-#FROM alpine:3.10
-# debian because build with dynamic linking with debian:buster
-FROM arm64v8/debian:buster-slim as lemmy
-
-#COPY --from=qemu /usr/bin/qemu-arm-static /usr/bin
-COPY --from=qemu /usr/bin/qemu-aarch64-static /usr/bin
-
-# Install libpq for postgres
-#RUN apk add libpq
-RUN apt-get update && apt-get install -y libpq5
-
-RUN addgroup --gid 1000 lemmy
-# for alpine
-#RUN adduser -D -s /bin/sh -u 1000 -G lemmy lemmy
-# for debian
-RUN adduser --disabled-password --shell /bin/sh --uid 1000 --ingroup lemmy lemmy
-
-# Copy resources
-COPY server/config/defaults.hjson /config/defaults.hjson
-COPY --from=rust /app/server/ready /app/lemmy
-COPY --from=node /app/ui/dist /app/dist
-
-RUN chown lemmy:lemmy /app/lemmy
-USER lemmy
-EXPOSE 8536
-CMD ["/app/lemmy"]
diff --git a/docker/dev/Dockerfile.armv7hf b/docker/dev/Dockerfile.armv7hf
deleted file mode 100644
index c2c9084c..00000000
--- a/docker/dev/Dockerfile.armv7hf
+++ /dev/null
@@ -1,79 +0,0 @@
-FROM node:10-jessie as node
-
-WORKDIR /app/ui
-
-# Cache deps
-COPY ui/package.json ui/yarn.lock ./
-RUN yarn install --pure-lockfile
-
-# Build
-COPY ui /app/ui
-RUN yarn build
-
-
-# contains qemu-*-static for cross-compilation
-FROM multiarch/qemu-user-static as qemu
-
-
-FROM arm32v7/rust:1.37-buster as rust
-
-#COPY --from=qemu /usr/bin/qemu-aarch64-static /usr/bin
-COPY --from=qemu /usr/bin/qemu-arm-static /usr/bin
-
-
-# Install musl
-#RUN apt-get update && apt-get install -y mc
-#RUN apt-get install -y musl-tools mc
-#libpq-dev mc
-#RUN rustup target add ${TARGET}
-
-# Cache deps
-WORKDIR /app
-RUN USER=root cargo new server
-WORKDIR /app/server
-COPY server/Cargo.toml server/Cargo.lock ./
-RUN  mkdir -p ./src/bin \
-  && echo 'fn main() { println!("Dummy") }' > ./src/bin/main.rs
-#RUN cargo build --release
-# RUN cargo build
-  RUN RUSTFLAGS='-Ccodegen-units=1' cargo build
-COPY server/src ./src/
-COPY server/migrations ./migrations/
-RUN rm -f ./target/release/deps/lemmy_server* ; rm -f ./target/debug/deps/lemmy_server*
-
-
-# build for release
-#RUN cargo build --frozen --release
-RUN cargo build --frozen
-
-# Get diesel-cli on there just in case
-# RUN cargo install diesel_cli --no-default-features --features postgres
-
-RUN cp /app/server/target/debug/lemmy_server /app/server/ready
-#RUN cp /app/server/target/release/lemmy_server /app/server/ready
-
-#FROM alpine:3.10
-# debian because build with dynamic linking with debian:buster
-FROM arm32v7/debian:buster-slim as lemmy
-
-COPY --from=qemu /usr/bin/qemu-arm-static /usr/bin
-
-# Install libpq for postgres
-#RUN apk add libpq
-RUN apt-get update && apt-get install -y libpq5
-
-RUN addgroup --gid 1000 lemmy
-# for alpine
-#RUN adduser -D -s /bin/sh -u 1000 -G lemmy lemmy
-# for debian
-RUN adduser --disabled-password --shell /bin/sh --uid 1000 --ingroup lemmy lemmy
-
-# Copy resources
-COPY server/config/defaults.hjson /config/defaults.hjson
-COPY --from=rust /app/server/ready /app/lemmy
-COPY --from=node /app/ui/dist /app/dist
-
-RUN chown lemmy:lemmy /app/lemmy
-USER lemmy
-EXPOSE 8536
-CMD ["/app/lemmy"]
diff --git a/docker/dev/Dockerfile.libc b/docker/dev/Dockerfile.libc
deleted file mode 100644
index 6348342f..00000000
--- a/docker/dev/Dockerfile.libc
+++ /dev/null
@@ -1,88 +0,0 @@
-# can be build on x64, arm32, arm64 platforms
-# to build on target platform run 
-# docker build -f Dockerfile.libc -t dessalines/lemmy:version ../..
-#
-# to use docker buildx run
-# docker buildx build --platform linux/amd64,linux/arm64 -f Dockerfile.libc -t YOURNAME/lemmy --push ../..
-
-FROM node:12-buster as node
-# use this if use docker buildx 
-#FROM --platform=$BUILDPLATFORM node:12-buster as node
-
-WORKDIR /app/ui
-
-# Cache deps
-COPY ui/package.json ui/yarn.lock ./
-RUN yarn install --pure-lockfile --network-timeout 100000
-
-# Build
-COPY ui /app/ui
-RUN yarn build
-
-
-FROM rust:1.42 as rust
-
-# Cache deps
-WORKDIR /app
-
-RUN USER=root cargo new server
-WORKDIR /app/server
-COPY server/Cargo.toml server/Cargo.lock ./
-RUN  mkdir -p ./src/bin \
-  && echo 'fn main() { println!("Dummy") }' > ./src/bin/main.rs
-
-
-RUN cargo build --release
-#RUN cargo build && \
-#    rm -f ./target/release/deps/lemmy_server* ; rm -f ./target/debug/deps/lemmy_server*
-COPY server/src ./src/
-COPY server/migrations ./migrations/
-
-
-# build for release
-# workaround for https://github.com/rust-lang/rust/issues/62896
-#RUN RUSTFLAGS='-Ccodegen-units=1' cargo build --release
-RUN cargo build --release --frozen
-#RUN cargo build --frozen
-
-# Get diesel-cli on there just in case
-# RUN cargo install diesel_cli --no-default-features --features postgres
-
-# make result place always the same for lemmy container
-RUN cp /app/server/target/release/lemmy_server /app/server/ready
-#RUN cp /app/server/target/debug/lemmy_server /app/server/ready
-
-
-FROM rust:1.42 as docs
-
-WORKDIR /app
-
-# Build docs
-COPY docs ./docs
-RUN cargo install mdbook
-RUN mdbook build docs/
-
-
-#FROM alpine:3.10
-# debian because build with dynamic linking with debian:buster
-FROM debian:buster as lemmy
-
-# Install libpq for postgres
-#RUN apk add libpq
-RUN apt-get update && apt-get install -y libpq5
-RUN addgroup --gid 1000 lemmy
-# for alpine
-#RUN adduser -D -s /bin/sh -u 1000 -G lemmy lemmy
-# for debian
-RUN adduser --disabled-password --shell /bin/sh --uid 1000 --ingroup lemmy lemmy
-
-# Copy resources
-COPY server/config/defaults.hjson /config/defaults.hjson
-COPY --from=node /app/ui/dist /app/dist
-COPY --from=docs /app/docs/book/ /app/dist/documentation/
-COPY --from=rust /app/server/ready /app/lemmy
-
-RUN chown lemmy:lemmy /app/lemmy
-USER lemmy
-EXPOSE 8536
-CMD ["/app/lemmy"]
diff --git a/docker/dev/docker-compose.yml b/docker/dev/docker-compose.yml
index 3fc94099..bdcb4308 100644
--- a/docker/dev/docker-compose.yml
+++ b/docker/dev/docker-compose.yml
@@ -1,15 +1,6 @@
 version: '3.3'
 
 services:
-  postgres:
-    image: postgres:12-alpine
-    environment:
-      - POSTGRES_USER=lemmy
-      - POSTGRES_PASSWORD=password
-      - POSTGRES_DB=lemmy
-    volumes:
-      - ./volumes/postgres:/var/lib/postgresql/data
-    restart: always
 
   lemmy:
     build: 
@@ -22,17 +13,28 @@ services:
       - RUST_LOG=debug
     volumes:
       - ../lemmy.hjson:/config/config.hjson
-    depends_on:
+    depends_on: 
+      - pictrs
       - postgres
-      - pictshare
       - iframely
 
-  pictshare:
-    image: hascheksolutions/pictshare:latest
-    ports:
-      - "127.0.0.1:8537:80"
+  postgres:
+    image: postgres:12-alpine
+    environment:
+      - POSTGRES_USER=lemmy
+      - POSTGRES_PASSWORD=password
+      - POSTGRES_DB=lemmy
+    volumes:
+      - ./volumes/postgres:/var/lib/postgresql/data
+    restart: always
+
+  pictrs:
+    image: asonix/pictrs:v0.1.13-r0
+    ports: 
+      - "127.0.0.1:8537:8080"
+    user: 991:991
     volumes:
-      - ./volumes/pictshare:/usr/share/nginx/html/data
+      - ./volumes/pictrs:/mnt
     restart: always
 
   iframely:
diff --git a/docker/dev/docker_update.sh b/docker/dev/docker_update.sh
index 9d0f4542..21e92ba8 100755
--- a/docker/dev/docker_update.sh
+++ b/docker/dev/docker_update.sh
@@ -1,2 +1,6 @@
 #!/bin/sh
+set -e
+
+export COMPOSE_DOCKER_CLI_BUILD=1
+export DOCKER_BUILDKIT=1
 docker-compose up -d --no-deps --build
diff --git a/docker/dev/test_deploy.sh b/doc