>>HDAMDSZB (OP) wtf is kyogi?
No real organization....
here is what the current federation map looks like i drew this by downloading the following json of every board of every known instance (poopchan seems to be down). each instance has a color (above it) and each colored line mean the respective instance is pulling. i will look into a less retarded way to draw a graph. if you know one, leave a suggestion
>>HDAMDSZB (OP) very good visualisation thank you OP I just find it a bit messed up that boards can get federated to boards with different names sure, it makes sense from API standpoint, so that you can federate /g/ and /tech/, but don't federate irrelevant boards FFS
>>J95MM1A5 added poopchan. i had to define the vertexes and edges manually, but since that fitchan was created the map has changed (penchan, for example, created /fit/ and /out/ to federate with them) so i will now work on a way to automatically fetch and render the network map. should not be too hard. >famous last words
>>ZGRJNDOB almost done, just figuring out how to do clusters. i might just do it adhoc ...later, because now i will sleep >>9W0OBQXR i edited the old file and you are right, looks much better (pic related)
>>ZWPOEJQ9 here it is, automatically generated the phantom arrows are there because whoever retard runs 2hu-ch.org is trying to pull from non-existent board on 0xchan (like 2hu-ch.org/prog trying to pull from 0x00000000.xyz/prog) i will look into completely excluding 0xchan (or any arbitrary instance) from the graph
>>fb-V7SHJZRY as i said on >>2RSSDNFY, i use graphiviz (specifically the dot filter) to draw the graphs everything else is done on shell with some python scripts to help i wrote scripts for: 1. downloading all known instances' frontpages [shell] 2. parsing the hmtl to get the board list of each instance [python] 3. downloading the following json of each board [shell] 4. parsing the json into graph edges in the dot syntax [python] 5. building the rest of the dot file (setting colors and text for each node, writing the header and creating the clusters) [shell] 6. calling all other scripts in the correct order [shell] i had never used python before, had seen too many examples of shitware written with it, but it is a decent scripting language, i guess soydevs just misuse it. i only chose it because it was the first result when i looked for an html parser and i didn't want to download new packages just for it (python already comes with the html and json parsers by default, it seems) >>9Z15ABYZ looks very pretty. i will clean up my code and post it somewhere so someone (possibly me) can convert it to gource
a few other remarks - there are loki instances federating now, i will later look into how to download data from them to properly include them on the map, any tips are welcome - given the direction this thread has taken, maybe it should be moved to /prog/ - is there any anonware license? something that requires it to be distributed anonymously (pseudonymously does not qualify)
>>K8ILZOJ1 Ok heres your feedback: Running as user returns: >curl: (7) Failed to connect to localhost port 9050: Connection refused >curl failed to download 'https://fchan.xyz/' Running as root returns: >picrel
>>36YN8I3Q read the DEPENDENCIES files, localhost port 9050 is where tor is supposed to listen, if you don't have tor running it will fail you obviously won't be able to download from kyogi, so you may need to delete its entry from output/known-instances (both the url and boardlist) if it causes the rest of the script to fail (i don't expect it to be very robust) >>J9D00PG0 a day has passed so i will take it down to do some maintenance i can put it back online if anyone cares
>>JXGLC70O >read the DEPENDENCIES files, localhost port 9050 is where tor is supposed to listen, if you don't have tor running it will fail you should just add an exception for tor if it's not running and run without it, to compensate for that >a day has passed so i will take it down to do some maintenance >i can put it back online if anyone cares why not leave it up? nothing wrong with an anonware site staying up for the lulz
>>2OERK7OY >you should just add an exception for tor if it's not running and run without it, to compensate for that i should get to that, the way i made the scripts is not robust at all and they will probably fail if anything is different from my machine i will add an option that needs to be passed in order for the script to try to reach onion services >why not leave it up? nothing wrong with an anonware site staying up for the lulz alright i will put it back up just for you when i am done here ;3 >i also dont have anything in the /data folder for some reason so nothing is being parsed at all. Not sure if i fucked something up. it is probably due to the failure you mentioned before. delete the '--proxy socks5h://localhost:9050' from the curl alias (@data-fetcher:5) and try again. data/ contains the downloaded files, and if the very first curl call failed it is no surprise that nothing was downloaded >did extracting with tar instead of pax fuck something up? i highly doubt it, if it unpacked wrong you should have gotten mangled text or something obviously corrupted, lossless compression usually fails catastrophically when it does i think i will start to use git to track changes, how is that cute git html interface called? gitweb was it? i will try to set that up.
>>HDAMDSZB (OP) up again http://p6nhckzlonbw72mhxqcyfa474ssavlnud6tvpmhjzf37r2zyz2ommtqd.onion/projects/fcham/ now will only use tor if the -t option is passed to data-fetcher and if a download fails it will just skip it instead of aborting (except for fchan.xyz, because it is needed to get the instance list) and it has proper option handling with getopt i am not sure if i should bother with an http git frontend or if just having a basic git server will suffice. i will try the basic one first
>>7WHVFRU0 >i am not sure if i should bother with an http git frontend or if just having a basic git server will suffice as long as you know how git works a basic server will be fine, unless a git frontend is something you want for a personal project
>>3ZXLZLIH it is done, now you can clone with git -c http.proxy=socks5h://localhost:9050 clone http://git.p6nhckzlonbw72mhxqcyfa474ssavlnud6tvpmhjzf37r2zyz2ommtqd.onion/fcham.git (needs the tor daemon running) stagit let's me specify a clone url and advertises it on the pages with the git clone command, but i had to edit the source to make it say that the -c option is needed as well. thankfully it is readable. a couple of things i still want to do: update the html everytime i push (it is static, after all) and figure a way to accept anonymous code contributions. maybe syntax highlighting would be nice to have on the files, but not if it is too much trouble to set up.
i am gonna have to gt this one >try to ssh into server >no-route-to-host.txt >go to computer hosting server >frozen >hold power to poweroff >turn back on >tsunami_siren.wav >no idea what the beeping means >open it up >pull out ram stick >turn on >other beep pattern >put ram back on, not sure what is happening >try to turn it on again >it works i wonder if my computer is dying this is why you shouldn't host from home, kids
>>IFLXUQWW not bad, the color scheme could be improved a bit (https://kill-9.xyz) this site has a good scheme with the css listed somewhere on it. other than that it looks good also you should make a post on /prog/ about this so people can find it, rather than digging through this thread
>>Y9IIWTWP >https://kill-9.xyz funny you mention them, my css is actually based on their darkgreen theme >also you should make a post on /prog/ i will, but i want to figure out a way to receive contributions first while researching it i read that projects used to be maintained with mailing lists where people would share their diffs and discuss changes, instead of using git-specific platforms like github or gitlab the pro git book explains it >https://git-scm.com/book/en/v2/Appendix-C%3A-Git-Commands-Email >https://git-scm.com/book/en/v2/Distributed-Git-Maintaining-a-Project >https://git-scm.com/book/en/v2/Distributed-Git-Contributing-to-a-Project#_public_project the last link caught my eye, it has a way to do pull requests that i could use, basically: >clone my repo >create a new local branch >work on your changes >push to a repo on your own website (inb4 internet pleb) >create a pull request >git request-pull origin/master yourrepo >yourrepo is a remote repo, where i will later pull your changes from (hence "pull" request) >origin/master is a pointer to the commit where the diffs will be calculated >pass -p to output the patch, instead of just a diff summary >send me the output of the command (i am still figuring this part out, maybe just post it on the thread) >i pull from your repo >merge with my repo's master >contribution has been incorporated, your name (which i recommend you configure to be "Anonymous") will show up as the author of your commits if i understood it right, that's the gist of it you will just need a place to host your clone (a repo hosting service might suffice, but why not set up your own stuff like me?) and i will need a way to receive a notice that someone requested a pull, other than this, it seems pretty straight forward to me
>>V4DWDHVU if you can't host a repo, you can also >clone my repo >create a new local branch >work on your changes >git diff origin/master >send me the output it will generate a diff of all the changes you made that i can then patch onto my repo, but there is no commit history with it, i think
>>ITH4SAN5 no prob, i hope we can collaborate soon, let me know if you need help setting up git i have been thinking of also setting up an irc server to have faster discussions about the project, seems easy enough. i heard i should have ssl for irc even over tor, but i didn't see any good reason for it, seems redundant to me idk
>>GW1KBZLT fuck nvm that ftp was fine on LAN, but it is all fucky over tor i am not sure what the problem is, but it seems the protocol itself is a bitch to work with tor because the ftpd will try to open a random socket for the data transfers, but to host an onion service i need to manually map external ports to internal ports (basically whitelist them) i even tried to configure the range of ports the ftpd will use and it still didn't work. from what i saw on the error messages, i am trying to connect to myself (localhost) and that is why it is failing, so it means the server sends the ip address the client is supposed to connect to (instead of doing the sane thing and just using the same connection) which in this case is 127.0.0.1 because that is the ip the daemon on the server binds to idk if i got this wrong or if ftp is really this retarded i wonder if there is another protocol that was made specifically for only receiving files, i don't need all the functionalities of ftp for my purposes
>>QBT8NSA4 connecting and sending ftp commands to the server works alright, apparently, is it the data connection (ftp has the brilliant idea of using 2 connections, one for commands, one for data) that is broken at this point i am considering rolling my own autistic solution using netcat, but i will look into alternatives, maybe study more about ftp to see how it works and what exactly is going wrong
i have just finished translating follow-parser from python to lua. i found a json parser for lua that is written in c that is ridiculously faster than python's. i had a little trouble translating because lua doesn't use posix/perl regex, i thought it was bullshit but then i read their justification (picrel) >imagine being so based that you disregard posix because it is bloated fcham used to hang for a few seconds on "==> generating graph", but now it breezes through it anyway, i might end up translating other scripts to lua, because: 1. i want to remove the python dependency, so translating board-parser is necessary 2. graph-maker would probably work better and be easier to read if it was written in a proper programming language (not to shit on shell, i was surprise i could do so much with it) and translating it to lua would allow me to merge it with follow-parser which will reduce code complexity because i won't have to maintain two color lists, for example
>>3IGM31FT on another note: i have changed the framework i use to generate my site's pages. i used to write the html manually, but i have now adapted (read: severely simplified) suckless' build-page.c tool to allow me to write my stuff in markdown. the appearance of the pages hasn't changed, but now is my way comfier to write them. i say this because after seeing how fairly easy it is to automatically generate static html, and having taken a look at the fchan's board outbox json, i might be willing to try to create a simple fchan client that let's you subscribe to different boards (download their outboxes) and renders a static html page that you can open on your browser. you won't be able to post replies (at least i have no idea how to implement that), but it can link directly to an instance's page so you can reply from there if you want. i don't intend to make it awfully complicated, so no sql fuckery, just something like fcham: download the json and parse it. it may well download attachments locally and avoid redownloading them i want to come up with a very clever name for it though
>>HD0OP9A9 i am satisfied with the state of fcham so i finally made the thread on /prog/ (>>fprog-OFF4C8HU →), here are the highlights of the changes i have made since my last post * follow-parser and graph-maker have been merged, they are now a lua script * board-parser is now just a simple shell pipeline inside data-fetcher, the html parsing was overkill * python is no longer a dependency * lua is, and while it has some dependencies (json parser and getopt), they are included in the repo for convenience * data-fetcher no longer overwrites data by default, if you want it to, pass the -o flag * data-fetcher has a option to redownload a single instance, use the -r flag specifying the instance's domain name * instead of a fcham master script that calls all other scripts, the scripts are now organized by make recipes, a simple 'make' does everything * defining clusters has been greaty simplified, it basically just works now and is much easier to configure which types of board are clustered together (like /b/ and /vip/) * speaking of configure, it now has a configuration file, you can define an ignorelist (instances to hide in the graph), a mergelist, whether to show unknown instances, and the colors of the nodes using a configuration file. an example file is provided, if you want to use it copy it to ~/.config/fcham/config.lua, or pass the -f flag to graph-maker to define a custom location graph-maker and data-fetcher have been renamed to a verb-name (make-graph and fetch-data)