Web Scraping On Java 17 & Htmlunit

We use the capabilities of java 8+ (17) and htmlunit library to write our own web-parser.

What you'll learn

You will create a Java Project

Follow me material feed

Suitable for beginner programmers but better to be a slightly experienced

From scratch to release - will install the environment, write the code and build the jar - step by step.

Will make some tests, touch multithreading, apply design patterns and use java 8+ syntax

Requirements

Computer skills. Software installation skills.

Some java programming experience and understanding of XML/JSON/HTML are preferred

Description

This is a "Follow me" product development course.In this project you will touch on:Java 8+, lambda expressions, varMaven - modification, assembly of the jar archive and its subsequent launchXPath - and its syntaxTestingMultithreadingApplications of Singleton and Factory patternsworking with regular expressionsworking with Gson and saving a projectAnd we won't use Spring just out of spite)The project begins with Intelliji IDEA Community installation, deploying the base project on Maven and till the jar build so that the program can work independently without an IDE (development environment). The parsing result will written as a json files to a hostnamed forlder.If you understand at least some of the words and if you are interested in your own website parsing project, this course is for you.I set the entry level for this course as an intermediate level and at the same time kept the entry level in mind. At the end, the boundaries between entry-level and intermediate levels are largely blurred, and a person considering his level to be average and with experience in a company can “skip” some initial knowledge and vice versa.In this course, I apply work experience training new developers, where we work through a task together, conveying not only an understanding of the design, but also the design approach and semantics.Although the training is built on the principle of do as I'm, I still think it is important to explain what is happening on the screen. If it's not enough, I'll add it. Let's get started.

Overview

Section 1: Введение

Lecture 1 IntelliJ IDEA Community Edition

Lecture 2 Project structure

Lecture 3 Deploying the project from Maven

Lecture 4 Maven Dependency

Lecture 5 Web site analyze

Section 2: Web site parsing

Lecture 6 ParseService

Lecture 7 ParseFactory - use design pattern to simplify code maintain

Lecture 8 Fetch pattern by URL

Lecture 9 Singleton approach

Lecture 10 Scheduler run

Lecture 11 More about XPath

Lecture 12 Parse list of data

Lecture 13 Build single jar

Lecture 14 Save results to a JSON file

Section 3: More info

Lecture 15 Parsing issues

For beginners and those who have already started Java developers.