Compare commits

..

No commits in common. "master" and "v0.1.1" have entirely different histories.

19 changed files with 120 additions and 444 deletions

View File

@ -1,10 +0,0 @@
Dockerfile
CHANGELOG.md
CODE_OF_CONDUCT.md
Rakefile
test/
vendor/
vendor
test
*.gem
.git

3
.github/FUNDING.yml vendored
View File

@ -1,3 +0,0 @@
ko_fi: captn3m0
liberapay: captn3m0
github: captn3m0

2
.gitignore vendored
View File

@ -8,5 +8,3 @@
/tmp/
/vendor/
Gemfile.lock
*.gem
.env

View File

@ -7,39 +7,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## 1.0.2 - 2024-03-07
### Changed
- Dependency updates
- Fixes for new Outline API
## 1.0.1 - 2020-04-26
### Changed
- Dependency Updates
## 1.0.0 - 2020-04-26
### Changed
- Changes `push` to `sync`.
### Added
- `sync` now actually syncs the git repository, and maintains history.
## 0.2.2 - 2019-08-14
- Adds a `push` command (See #2)
## 0.2.0 - 2019-08-12
- Adds export command (See [#1013](https://github.com/outline/outline/pull/1013) for corresponding Outline PR)
- Adds support for running import|export via Docker
- Sets up `outline-export` and `outline-import` as rubygem executables
- Dockerfile
## 0.1.1 - 2019-07-24
@ -50,3 +18,5 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- Initial release
[unreleased]: https://github.com/olivierlacan/keep-a-changelog/compare/v0.1.0...HEAD

View File

@ -1,18 +0,0 @@
FROM ruby:3.3-alpine3.19
RUN apk add --no-cache git openssh-client rsync && \
echo -e "StrictHostKeyChecking no" >> /etc/ssh/ssh_config && \
mkdir /root/.ssh
WORKDIR /outliner
COPY . /outliner/
RUN echo "gem: --no-ri --no-rdoc" > ~/.gemrc && \
apk add --no-cache alpine-sdk && \
gem update --system && \
gem install bundler && \
bundle install && \
apk del --no-cache alpine-sdk && \
rm ~/.gemrc
ENTRYPOINT ["/outliner/entrypoint.sh"]

View File

@ -1,7 +1,7 @@
Copyright 2020 Abhay Rana
Copyright 2019 Abhay Rana
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

View File

@ -1,6 +1,6 @@
# Outliner [![Gem Version](https://badge.fury.io/rb/outliner.svg)](https://badge.fury.io/rb/outliner) [![Docker Cloud Build Status](https://img.shields.io/docker/cloud/build/captn3m0/outliner)](https://hub.docker.com/r/captn3m0/outliner) [![Docker Image Version (latest semver)](https://img.shields.io/docker/v/captn3m0/outliner)](https://hub.docker.com/r/captn3m0/outliner) [![Docker Image Size (latest semver)](https://img.shields.io/docker/image-size/captn3m0/outliner)](https://hub.docker.com/r/captn3m0/outliner)
# Outliner [![Gem Version](https://badge.fury.io/rb/outliner.svg)](https://badge.fury.io/rb/outliner)
A simple HTTParty based wrapper for the [Outline API](https://www.getoutline.com/developers). It also offers a one-line import option to let you migrate an existing set of Markdown files to Outline. For quickly running export/import commands, you can use the Docker Image as well.
A simple HTTParty based wrapper for the [Outline API](https://www.getoutline.com/developers). It also offers a one-line import option to let you migrate an existing set of Markdown files to Outline.
## Installation
@ -28,18 +28,11 @@ require 'outliner'
client = Outliner.new('https://knowledge.example.com')
pp client.auth_info
pp client.collections_list(offset: 0, limit: 10)
# This works around a 302 redirect bug in httparty
begin
r = @client.fileOperations__redirect({id: FILE_OPERATION_ID}, format: nil, no_follow: true)
rescue HTTParty::RedirectionTooDeep => e
# Download this using response = HTTParty.get e.response.header['location'] if needed
pp e.response.header['location']
end
```
### Import
`outliner` can be used to import an existing collection of documents into Outline. To do this, run:
`outliner` can be used to import an existing collection of documents into Outline. To do this run:
```bash
export OUTLINE_BASE_URI="https://kb.example.com"
@ -47,71 +40,13 @@ export OUTLINE_TOKEN="PUT YOUR TOKEN HERE"
export SOURCE_DIRECTORY="/home/user/wiki"
export DESTINATION_COLLECTION_NAME="Archive"
bundle install outliner
outliner-import "$SOURCE_DIRECTORY" "$DESTINATION_COLLECTION_NAME"
```
### Export
`outliner` can be used to run a one-time export of all documents in Outline to a local directory. To do this, run:
```bash
export OUTLINE_BASE_URI="https://kb.example.com"
export OUTLINE_TOKEN="PUT YOUR TOKEN HERE"
# Ensure that this exists and is writable
export DESTINATION_DIRECTORY="/data"
bundle install outliner
outliner-export "$DESTINATION_DIRECTORY"
```
## Docker
You can use the pre-built docker image to run the above commands as well. See the following commands for examples:
### Setup
Copy the `env.sample` file to `.env` and update the values there.
### Export
Downloads all collections from Outline, and exports them as nested markdown files inside the given directory (`/data` inside the container, mount it accordingly.)
```bash
docker run --env-file .env
--volume /tmp:/data \
captn3m0/outliner \
export \
/data
```
### Import
Imports all markdown documents in a directory to a named Collection on outline. Creates the collection if it doesn't exist.
```bash
docker run --env-file .env
--volume /path/to/wiki:/data \
captn3m0/outliner \
import /data "Archive"
```
### Sync
Does a export from Outline, and pushes the corresponding result to the Git repository. Currenly does a force-push to the repository. Use with care.
Note: Sync is currently only available as a Docker Command
```bash
docker run --env-file .env
--volume /etc/ssh/private.key:/root/.ssh/id_rsa
captn3m0/outliner \
sync
bundle exec bin/import "$SOURCE_DIRECTORY" "$DESTINATION_COLLECTION_NAME"
```
#### Limitations
- [import] Images are currently not imported. Host them externally for this to work.
- [import] Only `.md` files are currently supported
- [docker] `StrictHostKeyChecking` is currently disabled for `push`, please only run this in trusted networks.
- Images are currently not imported
- Only `.md` files are currently supported
## Development

78
bin/import Executable file
View File

@ -0,0 +1,78 @@
#!/usr/bin/env ruby
require "bundler/setup"
require "outliner"
def validate
unless (ARGV.size == 2) and Dir.exists?(ARGV[0]) and ARGV[1].match(/\w+/) and ENV.key?('OUTLINE_BASE_URI')
puts "[E] Please call as import local_directory remote_collection_name"
puts "[E] Please export OUTLINE_BASE_URI and OUTLINE_TOKEN environment variables"
puts "[E] OUTLINE_BASE_URI should not include /api"
end
end
def find_or_create_collection(name)
collections = CLIENT.collections_list(limit: 100)['data']
collections.filter!{|c|c['name'] == name}
if collections.size >= 1
collections[0]['id']
else
CLIENT.collections_create(name: name, description: 'Imported Collection')['data']['id']
end
end
def create_documents_recursively(directory, collection_id, parent_document_id=nil)
cwd = Dir.pwd
Dir.chdir directory
# Create all documents for this directory
Dir["*.md"].each do |file|
params = {
title: file[0...-3],
text: file[0...-3] + "\n" + File.read(file) + "\n\n---\nImported at #{Time.now}",
collectionId: collection_id,
publish: true
}
params[:parentDocumentId] = parent_document_id if parent_document_id
CLIENT.documents_create(params)
puts "[-] #{file}"
end
# Create child documents for each sub-directory
Dir.glob('*').select {|f| File.directory? f}.each do |dir|
puts "[-] #{dir}"
params = {
title: dir,
text: dir +"\nImported at #{Time.now}",
collectionId: collection_id,
publish: true,
parentDocumentId: parent_document_id
}
response = CLIENT.documents_create(params)
create_documents_recursively(dir, collection_id, response['data']['id'])
end
Dir.chdir cwd
end
# Run validations
validate
# Setup variables
local_directory = ARGV[0]
remote_collection_name = ARGV[1]
# Create a root collection
CLIENT = Outliner::Client.new ENV['OUTLINE_BASE_URI']
root_collection_id = find_or_create_collection(remote_collection_name)
begin
create_documents_recursively(local_directory, root_collection_id)
puts "[S] Import successful"
rescue Exception => e
# If we fail, print an error, and delete the collection
puts "[E] Import failed with error: #{e.message}"
CLIENT.collections_delete(id: root_collection_id)
puts "[E] Deleted collection, please report the issue or retry"
exit 1
end

View File

@ -1,79 +0,0 @@
#!/bin/sh
set -eu
if [ $# -eq 0 ]; then
echo "Please run with outliner [export|import|sync] arguments"
exit
fi
setup_git() {
if [ -f "$HOME/.ssh/id_rsa" ]; then
# This is required because Kubernetes secret mounts can't
# have file permissions set
chmod 0400 "$HOME/.ssh/id_rsa"
if [ ! -d "$HOME/.ssh/id_rsa.pub" ]; then
ssh-keygen -y -f "$HOME/.ssh/id_rsa" > "$HOME/.ssh/id_rsa.pub"
fi
echo "[+] Using SSH key for git pushes"
else
echo "[E] Git credentials not available, quitting"
exit 1
fi
eval $(ssh-agent)
ssh-add "$HOME/.ssh/id_rsa"
}
update_git_config() {
EMAIL=${GIT_EMAIL:-outliner@example.invalid}
git config --global user.email "$EMAIL"
git config --global user.name "Outliner Backup"
git remote add origin "$GIT_REMOTE_URL"
}
case $1 in
export)
shift
bundle exec outliner-export "$@"
;;
import)
shift
bundle exec outliner-import "$@"
;;
sync)
BRANCH=${GIT_BRANCH:-master}
old_git_dir=$(mktemp -d)
fresh_export_dir=$(mktemp -d)
if [ -z "$GIT_REMOTE_URL" ]; then
echo "[E] GIT_REMOTE_URL not set"
exit 1
else
git clone --branch "$BRANCH" "$GIT_REMOTE_URL" "$old_git_dir"
setup_git
update_git_config
echo "[+] Exporting data from Outline"
bundle exec outliner-export "$fresh_export_dir"
echo "[+] Resetting git repository"
cd "$old_git_dir"
# We update so that git forgets all files
git ls-files -z |xargs -n1 -0 git rm
# Then we copy across the files from the new export
cd "$fresh_export_dir"
echo "[+] Updating git repository"
rsync -av . "$old_git_dir"
cd "$old_git_dir"
echo "[+] Committing to git"
git add .
git commit --message "Backup: $(date)" > /dev/null
git status
echo "[+] Pushing to git remote"
git push origin "HEAD:$BRANCH"
echo "[+] Cleaning up"
rm -rf "$old_git_dir"
rm -rf "$fresh_export_dir"
fi
;;
*)
echo "Invalid command, please check README"
;;
esac

View File

@ -1,4 +0,0 @@
OUTLINE_BASE_URI=https://outline.example.com
GIT_BRANCH=master
GIT_REMOTE_URL=git@example.com:org/outline.backup.git
OUTLINE_TOKEN=

View File

@ -1,49 +0,0 @@
#!/usr/bin/env ruby
# frozen_string_literal: true
require 'bundler/setup'
require 'outliner'
require 'tempfile'
def validate
raise 'Missing arguments' if ARGV.size != 1
raise 'Invalid directory' unless Dir.exist?(ARGV[0])
raise 'OUTLINE_BASE_URI not set' unless ENV.key?('OUTLINE_BASE_URI')
raise 'OUTLINE_TOKEN not set' unless ENV.key?('OUTLINE_TOKEN')
end
# Run validations
validate
# Setup variables
local_directory = ARGV[0]
CLIENT = Outliner::Client.new ENV['OUTLINE_BASE_URI']
# Download the complete zip
response = CLIENT.collections__export_all(format: "outline-markdown")
raise 'Failed to trigger export_all action' if not response['success']
file_operation_id = response['data']['fileOperation']['id']
fop_info_response = nil
i = 0
loop do
i += 1
raise 'Timed out waiting for the file export operation to complete' if i > 20
sleep(2*i)
fop_info_response = CLIENT.fileOperations__info(id: file_operation_id)
raise 'Failed to query export file operation info' if not fop_info_response['ok']
break if fop_info_response['data']['state'] == 'complete'
end
begin
fop_redirect_response = CLIENT.fileOperations__redirect({id: file_operation_id}, {no_follow: true})
rescue HTTParty::RedirectionTooDeep => e
response = HTTParty.get e.response.header['location']
file = Tempfile.new('download.zip')
File.open(file.path, 'w') { |f| f.write(response.body) }
`unzip -o "#{file.path}" -d "#{local_directory}"`
file.unlink
end

View File

@ -1,71 +0,0 @@
#!/usr/bin/env ruby
# frozen_string_literal: true
require 'bundler/setup'
require 'outliner'
def validate
raise 'Missing arguments' if ARGV.size != 2
raise 'Invalid directory' unless Dir.exist?(ARGV[0])
raise 'Invalid collection' unless ARGV[1].match(/\w+/)
raise 'OUTLINE_BASE_URI not set' unless ENV.key?('OUTLINE_BASE_URI')
raise 'OUTLINE_TOKEN not set' unless ENV.key?('OUTLINE_TOKEN')
end
def create_documents_recursively(directory, collection_id, parent_document_id = nil)
cwd = Dir.pwd
Dir.chdir directory
# Create all documents for this directory
Dir['*.md'].each do |file|
params = {
title: file[0...-3],
text: file[0...-3] +
"\n" +
File.read(file) +
"\n\n---\nImported at #{Time.now}",
collectionId: collection_id,
publish: true
}
params[:parentDocumentId] = parent_document_id if parent_document_id
CLIENT.documents__create(params)
puts "[-] #{file}"
end
# Create child documents for each sub-directory
Dir.glob('*').select { |f| File.directory? f }.each do |dir|
puts "[-] #{dir}"
params = {
title: dir,
text: dir + "\nImported at #{Time.now}",
collectionId: collection_id,
publish: true,
parentDocumentId: parent_document_id
}
response = CLIENT.documents__create(params)
create_documents_recursively(dir, collection_id, response['data']['id'])
end
Dir.chdir cwd
end
# Run validations
validate
# Setup variables
local_directory = ARGV[0]
remote_collection_name = ARGV[1]
# Create a root collection
CLIENT = Outliner::Client.new ENV['OUTLINE_BASE_URI']
root_collection_id = CLIENT.find_or_create_collection(remote_collection_name)
begin
create_documents_recursively(local_directory, root_collection_id)
puts '[S] Import successful'
rescue StandardError? => e
# If we fail, print an error, and delete the collection
puts "[E] Import failed with error: #{e.message}"
CLIENT.collections__delete(id: root_collection_id)
puts '[E] Deleted collection, please report the issue or retry'
exit 1
end

View File

@ -1,5 +1,4 @@
require "outliner/version"
require "outliner/client"
module Outliner
class Error < StandardError; end

View File

@ -10,29 +10,18 @@ module Outliner
@token = ENV['OUTLINE_TOKEN']
end
def find_or_create_collection(name)
collections = self.collections__list(limit: 100)['data']
collections.filter!{|c|c['name'] == name}
if collections.size >= 1
collections[0]['id']
else
self.collections__create(name: name, description: 'Imported Collection')['data']['id']
end
end
def method_missing(method_name, params = {}, options = {})
method_name = "/#{method_name.to_s.sub('__', '.')}"
options = {
body: params.to_json,
def method_missing(method_name, params = {})
method_name = '/' + method_name.to_s.sub('_', '.')
body = {token: @token}.merge(params).to_json
options = {
body: body,
headers: {
'Accept' => 'application/json',
'Content-Type' => 'application/json',
'User-Agent' => "Outliner/#{Outliner::VERSION}",
'Authorization' => "Bearer #{@token}"
'Accept'=>'application/json',
'Content-Type': 'application/json',
'User-Agent': "Outliner/#{Outliner::VERSION}"
},
format: :json,
}.merge!(options)
format: :json
}
self.class.post(method_name, options)
end

View File

@ -1,3 +1,3 @@
module Outliner
VERSION = "1.0.2"
VERSION = "0.1.1"
end

View File

@ -2,13 +2,13 @@ lib = File.expand_path("lib", __dir__)
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
require "outliner/version"
require "outliner/client"
Gem::Specification.new do |spec|
spec.name = "outliner"
spec.version = Outliner::VERSION
spec.authors = ["Nemo"]
spec.email = ["outliner@captnemo.in"]
spec.licenses = ["MIT"]
spec.email = ["rubygem.outliner@captnemo.in"]
spec.summary = "A simple HTTParty based client for outline knowledge base."
spec.homepage = "https://github.com/captn3m0/outliner"
@ -17,17 +17,19 @@ Gem::Specification.new do |spec|
spec.metadata["source_code_uri"] = "https://github.com/captn3m0/outliner"
spec.metadata["changelog_uri"] = "https://github.com/captn3m0/outliner/blob/master/CHANGELOG.md"
spec.files = Dir['**/*'].reject { |f| f.match(%r{^(vendor|test|spec|features)/}) }
# Specify which files should be added to the gem when it is released.
# The `git ls-files -z` loads the files in the RubyGem that have been added into git.
spec.files = Dir.chdir(File.expand_path('..', __FILE__)) do
`git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
end
spec.bindir = "exe"
spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
spec.require_paths = ["lib"]
spec.add_dependency "httparty", "~> 0.21"
spec.add_dependency "json", "~> 2.7"
spec.add_dependency "httparty", "~> 0.17"
spec.add_development_dependency "bundler", "~> 2.1"
spec.add_dependency "rake", ">= 13.1"
spec.add_development_dependency "webmock", "~> 3.23"
spec.add_development_dependency "minitest", "~> 5.22"
spec.add_development_dependency "bundler", "~> 2.0"
spec.add_development_dependency "rake", "~> 10.0"
spec.add_development_dependency "webmock", "~> 3.6.0"
spec.add_development_dependency "minitest", "~> 5.8.4"
end

View File

@ -1,31 +0,0 @@
{
"success": true,
"data": {
"fileOperation": {
"id": "08d5db26-bf43-4ec9-ac62-8769fd828e94",
"type": "export",
"format": "outline-markdown",
"name": "Acme-export.zip",
"state": "creating",
"error": null,
"size": "0",
"collectionId": null,
"user": {
"id": "817fb131-4a9b-4981-9002-38c2503adc3e",
"name": "Acme Admin",
"avatarUrl": "https://fake-avatar-url.com",
"color": "#2BC2FF",
"isAdmin": true,
"isSuspended": false,
"isViewer": false,
"createdAt": "2024-03-07T04:03:45.204Z",
"updatedAt": "2024-03-07T06:51:26.023Z",
"lastActiveAt": "2024-03-07T06:51:26.023Z"
},
"createdAt": "2024-03-07T06:51:26.031Z",
"updatedAt": "2024-03-07T06:51:26.031Z"
}
},
"status": 200,
"ok": true
}

View File

@ -1 +0,0 @@
Redirecting to https://fake.s3-accelerate.amazonaws.com/uploads/3e11b7f9-f1c0-44d0-a21b-4d6e0561e9c9/a5b6985a-cff6-4d03-be60-20c517bee63e/Acme-export.zip?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=XXXXXXXXXXXXXXXXXXXX%2F20240307%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240307T055812Z&X-Amz-Expires=60&X-Amz-Signature=ff759b27ddfd5c7401c1715411a8ceba886f9f462c9b52fc0c4a5906e99ecd22&X-Amz-SignedHeaders=host&response-content-disposition=attachment.

View File

@ -6,7 +6,6 @@ require 'json'
class ClientTest < Minitest::Test
TOKEN = "c4302eFAKE_TOKEN9b6e27bccb7"
BASE_URI='https://kb.example.com'
FILE_OPERATION_ID = "08d5db26-bf43-4ec9-ac62-8769fd828e94"
def setup
ENV['OUTLINE_TOKEN'] = TOKEN
@client = Outliner::Client.new BASE_URI
@ -17,36 +16,9 @@ class ClientTest < Minitest::Test
end
def test_auth_info_api
mock('auth.info')
r = @client.auth__info
assert_equal "https://kb.example.com", r['data']['team']['url']
end
def test_export
mock('collections.export_all')
r = @client.collections__export_all
assert_equal FILE_OPERATION_ID, r['data']['fileOperation']['id']
assert_equal 200, r['status']
assert_equal true, r['ok']
end
def test_retrieve_file_operation
mock("fileOperations.redirect", {
id: FILE_OPERATION_ID
}, {
"X-Download-Options" => "noopen",
"X-Content-Type-Options" => "nosniff",
"Content-Type" => "text/plain; charset=utf-8",
"Content-Length" => "459",
"Location" => "https://s3.example.com/#{FILE_OPERATION_ID}"
}, 302)
begin
r = @client.fileOperations__redirect({id: FILE_OPERATION_ID}, format: nil, no_follow: true)
rescue HTTParty::RedirectionTooDeep => e
assert_equal "302", e.response.code
assert_equal "https://s3.example.com/#{FILE_OPERATION_ID}", e.response.header['location']
end
mock('auth.info', 'auth.info.200')
auth_info = @client.auth_info
assert_equal "https://kb.example.com", auth_info['data']['team']['url']
end
private
@ -55,16 +27,15 @@ class ClientTest < Minitest::Test
File.read "test/fixtures/#{file}.json"
end
def mock(method_name, params = {}, response_headers = {}, status = 200)
def mock(method_name, fixture_file, params = {})
stub_request(:post, BASE_URI + "/api/" + method_name)
.with(
body: params.to_json,
body: params.merge({token: TOKEN}).to_json,
headers: {
'Accept'=>'application/json',
'User-Agent'=>"Outliner/#{Outliner::VERSION}",
'Content-Type'=> 'application/json',
"Authorization"=> "Bearer #{TOKEN}"
'Content-Type'=> 'application/json'
}
).to_return(body: read_fixture(method_name + ".#{status}"), headers: response_headers, status: 302)
).to_return(body: read_fixture(fixture_file))
end
end
end