Translated by AI model Qwen/Qwen3-8B.
Source Language: Simplified Chinese, Target Language: english, Translation Time: 2026-05-01 14:43
.AI translation is for reference only. Accuracy is not guaranteed, please refer to the original text.
Interesting
The code is never finished, and the bugs are never fixed.
When I first got back from vacation, I updated the code for python-code/novel_crawler on Github, fixing several bugs, adding multi-threaded downloading, and creating a very nice command-line interface.
I always thought that multi-threaded downloading had been added a long time ago, but upon checking, I found out it was added recently. I remember I just randomly picked a novel website to test the code, and by the time I finished writing the code, I had probably sent tens of thousands of requests to that site. Now thinking back, I feel a bit guilty, as those tens of thousands of requests might have placed a significant burden on their server. However, the website's response speed was surprisingly fast, allowing me to crawl nearly 4,000 pages in 20 seconds.
I also want to add support for xpath to the python-code/novel_crawler project so that it can accurately extract content from more complex novel pages. But I haven't figured out how to maintain backward compatibility yet, and the changes would be quite extensive, so I've temporarily put it aside.
Later, I set up Actions for python-code, aiming to automatically package Python code into executable files using nuitka. To be honest, this process was quite frustrating. From having only heard of nuitka to becoming very familiar with it, the actions kept running and failing, sending me over seventy error emails ([God-2077/python-code] Run failed). I didn't understand how to configure them or write the relevant code. It was all about Git commits: action runs, errors occur, fix the code, commit again, action runs again, errors occur again, and so on...
But eventually, it worked, just needing some optimization and refinement.
In the end, as mentioned in the previous article, I went on to crawl websites and download books. After finishing the code for the previous article, I added a multi-threaded downloader on my phone that can automatically switch to Chinese and optimize EPUB images. However, since I wrote it on my phone, the code was kept as simple as possible, resulting in a messy structure, but it still ran.
Mobile phones are not suitable for reading books, the screen is too small.
I wrote the code on my phone because my dad took my computer. I probably won't be able to update the code until the vacation ends.
If you enjoyed this, leave a comment~