Automated YaCy Docker image

Forum for developers

Automated YaCy Docker image

Beitragvon luc » Sa Mai 14, 2016 9:18 am

Hi, I made last days some experiments using docker to deploy YaCy. I found it quite interesting as an alternative to full virtual machines : it offers complete isolation but is lightweight and image building can be easily shared and automated.
Don't you think it would be interesting to include a Dockerfile in main yacy git repository? It would allow a docker image up-to-date with latest commits to be automatically generated, on Docker Hub or on another Docker repository.
There were already 3 existing yacy docker files on Docker Hub, but they reference older YaCy versions or are not targetting automated build from latest sources.

I propose you a version based on Debian here : https://github.com/luccioman/yacy_searc ... ker/docker.
And the automated docker image : https://hub.docker.com/r/luccioman/yacy/
luc
 
Beiträge: 276
Registriert: Mi Aug 26, 2015 1:04 am

Re: Automated YaCy Docker image

Beitragvon Orbiter » Mo Mai 16, 2016 2:56 pm

Thats very interesting. It reminds me on latest activities at the YaCy sister-project loklak, which has now one-click deploy buttons for Heroku, Scalingo, Bluemix and the Docker Cloud:
https://github.com/loklak/loklak_server ... ick-deploy

It would be amazing to analyze if similar things are possible for YaCy. Can you try to copy the one-click button deploy schema of loklak for YaCy?
Orbiter
 
Beiträge: 5786
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Automated YaCy Docker image

Beitragvon luc » Di Mai 17, 2016 9:09 am

Yes that's great you set this up on loklak.
I will try it on YaCy as soon as I can.
luc
 
Beiträge: 276
Registriert: Mi Aug 26, 2015 1:04 am

Re: Automated YaCy Docker image

Beitragvon luc » Di Mai 17, 2016 9:28 pm

Currently the one-click deploy to docker cloud appears to be broken : the links works but then it opens an empty stack. However the link on loklak readme seems to follow documentation recommendation (https://docs.docker.com/docker-cloud/ap ... cloud-btn/)...
By the way I pushed a docker cloud stack for yacy (https://github.com/luccioman/yacy_searc ... -cloud.yml) which I successfully used to deploy a YaCy node on Docker Cloud.
I also added the button on my yacy docker readme, hoping this button should soon work.
luc
 
Beiträge: 276
Registriert: Mi Aug 26, 2015 1:04 am

Re: Automated YaCy Docker image

Beitragvon luc » Fr Mai 27, 2016 1:34 pm

@Orbiter, thank you for having merged Pull Request #52.
I don't know if you saw my comment after this request was close, so I copy it here :

Do you plan to create an official account for yacy organization on DockerHub?
If so YaCy docker image building could be fully automated from latest commits on main yacy git repository...
I am not member of yacy main github repository so I still have to manually merge latest commits to my own repository clone, wich then automatically triggers a docker build on https://hub.docker.com/r/luccioman/yacy/...
luc
 
Beiträge: 276
Registriert: Mi Aug 26, 2015 1:04 am

Re: Automated YaCy Docker image

Beitragvon Orbiter » Fr Mai 27, 2016 7:24 pm

ah, yes, good idea, but I believe Frank @frank_gamefreak has taken that account already :)
Frank?
Orbiter
 
Beiträge: 5786
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Automated YaCy Docker image

Beitragvon luc » Fr Jun 03, 2016 4:09 pm

Hi, in the meantime I continued playing with YaCy in Docker and I propose you some improvements, mainly regarding security and image size : see Pull Request #55 for more details.
luc
 
Beiträge: 276
Registriert: Mi Aug 26, 2015 1:04 am

Re: Automated YaCy Docker image

Beitragvon luc » So Jul 03, 2016 4:56 pm

After running continuously a YaCy Docker container for some weeks, I have some more informations for anyone interested :
- the way I configured the proposed YaCy image, with default Docker log configuration and a YaCy instance in peer to peer mode, the log file for the container grows up to some giga bytes in a week, eventually filling all disk on a small host server. The problem is not YaCy log which is correctly rotated, but Docker container log itself. To avoid this problem you have to set some options (--log-opt max-size and --log-opt max-file) when running the container. So I updated the Readme.md to mention this.
- I also added to the Readme.md an example procedure to upgrade the YaCy Docker image.
- Regarding Docker Hub automated build on commits, a simple daily scheduled shell script checking YaCy main git repository commits atom feed and then eventually triggering build through docker Hub Trigger URL seems to do the job.

@Orbiter, if you wish to merge these changes to main YaCy branch I think it could be a good idea before releasing next version as it has no impact on YaCy source code itself.
luc
 
Beiträge: 276
Registriert: Mi Aug 26, 2015 1:04 am

Re: Automated YaCy Docker image

Beitragvon Orbiter » Mo Jul 04, 2016 9:17 pm

I just merge the changes. I did not test them but I trust you.

In loklak we have four deploy-buttons, one with docker. It's possible to have a 1-click deploy in the could infrastructure of your choice. It would be nice to have that for YaCy as well. Can you have a look if that can be done for YaCy?
Orbiter
 
Beiträge: 5786
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Automated YaCy Docker image

Beitragvon luc » Mo Jul 04, 2016 9:36 pm

Great, I hope it will be useful for some people.

And of course it would be very convenient to have more one-click deploy possibilities for YaCy. I will try with loklak examples when having time in coming days and report here advancement. See u later.
luc
 
Beiträge: 276
Registriert: Mi Aug 26, 2015 1:04 am

Re: Automated YaCy Docker image

Beitragvon luc » Fr Jul 08, 2016 1:40 am

Hi, I have played with Heroku deploy first. You can from now try yourself the one-click deploy button for Heroku on this branch.

Thus with a free Heroku account you should be able to run a YaCy peer in senior mode (don't forget the only manual step : filling MAVEN_CUSTOM_OPTS environment variable with -Dyacy.staticIP=your_app_name.herokuapp.com, as explained in the open deploy page).

Remarks :
- I transformed main Readme file from mediawiki syntax to markdown as it is far easier and better documented to add button links in Markdown

- YaCy Jetty server http port has to be bound to the PORT arbitrarily chosen by Heroku, but external public port has to be set to 80, as your_app_name.herokuapp.com is the only way to reach an Heroku container from external machines. A small hack to serverSwitch.java do the job on the branch for now. But in yacy.init file there is a property named "bindPort", which looks like currently unused. It could be convenient to use it and to be able to define both a local bindPort and a public port in /Configbasic.html. What do you think about it?

- I am now faced with a big limitation of YaCy deployment in Heroku : files (Solr index, settings, logs...) are correctly written to container file system, but they are discarded each time the container (Heroku call it a "dyno") is restarted. This is an Heroku feature and this happens about once a day. So deploying in Heroku becomes quite handy for testing purpose, or for YaCy first try, but it is not very useful for long term use. Is it the same for loklak on Heroku or what is the storage solution? To overcome this limitation, it is documented that we should rely on cloud storage such as Amazon S3, but this would involve much refactoring, especially to keep compatibility with local file system storage for non-cloud peers. An intermediate solution could be to use Websolr Add-on to store solr index, but this is quite expensive (lowest price : 20 dollars a month for only 20000 documents!)...
luc
 
Beiträge: 276
Registriert: Mi Aug 26, 2015 1:04 am


Zurück zu YaCy Coding & Architecture

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 1 Gast