【英文】博客朋友圈

Posted on Edited on Views:

Preface

Implementing a blog friend circle on any blog platform.

Download the Project

  • It is recommended to download the project in a Linux environment. It has been tested that some dependencies cannot be downloaded and the program cannot run properly in a MacOS environment.
1
git clone https://github.com/Rock-Candy-Tea/hexo-circle-of-friends.git

Modify the Code

Customize the Crawling Rules

  • Since the crawler program does not have the crawling rules for the NexT theme, part of the code has been modified to add the crawling rules for the NexT theme. The actual crawling rules should be based on the structure of your own blog’s friends link page.

Define the variables to be crawled through the get_next_url() method: avatar, link, and name.
The method name can include a custom theme name get_xxx_url() method. When adding a global handler at the end of the method, set the fifth parameter to the theme name self.handle(avatar, link, name, queue, "xxx").
In the __init__() method, you need to add the configuration for the custom theme self.strategies = ("xxx").

hexo-circle-of-friends/hexo_circle_of_friends/utils/get_url.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
class GetUrl:

    def __init__(self):
        self.strategies = ("common1", "common2", "butterfly", "fluid", "matery", "nexmoe", "stun", "sakura", "volantis", "Yun", "stellar", "next")

    def get_next_url(self, response, queue):

        avatar = response.css(".link-grid .link-grid-container .link-grid-image::attr(data)").extract()

        link = response.css(".link-grid .link-grid-container a::attr(href)").extract()
        # Check if there is a 302 redirect prefix
        prefix = "/302.html?target="
        for i in range(len(link)):
            if link[i].startswith(prefix):
                link[i] = link[i][len(prefix):]

        name = response.css(".link-grid .link-grid-container p::text").extract()
        # Keep only the odd elements
        new_name_list = []
        for i in range(len(name)):
            if i % 2 == 0:
                new_name_list.append(name[i])
        name = new_name_list

        self.handle(avatar, link, name, queue, "next")

Configure Your Own Friends Link Page as the Starting Point for Crawling

If you customize the crawling rules, you need to set the theme to the custom theme name theme: "xxx".

hexo-circle-of-friends/hexo_circle_of_friends/fc_settings.yaml
1
2
3
LINK: [
     { link: "https://loli.fj.cn/links/", theme: "next" },
]

Deploy and Run Environment with Docker

  • Actually, it deploys the project runtime environment that has been configured by yyyzyyyz and then replaces the crawler project with the modified local project.

/root/hexo-circle-of-friends: the local path of the project

1
docker run -di --name circle -p 8000:8000 -v /tmp/:/tmp/ -v /root/hexo-circle-of-friends:/home/fcircle_src yyyzyyyz/fcircle:latest

Run the Crawler in Docker

1
docker exec circle nohup python3 -u ./hexo_circle_of_friends/run.py > /dev/null 2>&1 &

Run the Web Service in Docker

  • Run the web service in Docker to access the crawled data through the HTTP protocol
1
docker exec circle nohup python3 -u ./api/main.py > /dev/null 2>&1 &

Set up Cron Jobs

  • Set up a cron job to crawl every 30 minutes
1
crontab -e
1
*/30 * * * * docker exec circle nohup python3 -u ./hexo_circle_of_friends/run.py > ./log1.txt &

Frontend Rendering

  • Render data by asynchronously sending requests

In fact, this project includes the frontend deployment solution, but I feel that it doesn’t fit well with my current theme, so I wrote a minimalistic one myself.

<ip>: the IP and port of the server

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
<!doctype html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport"
          content="width=device-width, user-scalable=no, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <title>Friends Circle</title>
</head>
<body>

<h1>Friends Circle</h1>
<div id="app">
</div>

<script data-pjax="" type="module">
    async function init() {
        let response = await fetch("<ip>/all");
        let result = await response.json();
        
        for (const item of result["article_data"]) {
            let a = document.createElement("a");
            a.href = item["link"];
            a.classList.add("article_item");
            a.innerHTML = `
                    <div class="article_title">${item["title"]}</div>
                    <div class="article_author">${item["author"]}  Published at ${item["created"]}  Updated at ${item["updated"]}</div>
                    <br>
                `;
            document.getElementById("app").append(a);
        }

    }

    await init();

</script>

</body>
</html>

Done

References

Rock-Candy-Tea/hexo-circle-of-friends
Friends Circle Official Documentation