Apt Offline 1.8.2

Offline APT Package Manager

 Posted on February 9, 2020  |  19 minutes  |  3922 words  |  Ritesh Raj Sarraf

apt-offline 1.8.2

I am pleased to announce the release of apt-offline version 1.8.2

This release has many bug fixes along with a long standing issue of signature validation

2017 - The year of realization and change

Back in 2017, the bug was reported that apt-offline did not validate apt meta Packages files. apt-offline was only doing a signature validation for the Release files but did no validation of the apt meta Packages files, which had their checksums listed in the Release files. This validation was completely missing in apt-offline and gave the user the wrong impression that validation was in place.

I had hoped to fix this issue soon when it was reported, to have it part of the next Debian Stable release. But that never happened. On the contrary, I think 2 stable releases happened in between. And now it is 2020.

2017 was a year to spend a large chunk of my time on real life issues, for good. I realized that it is important to always give precedence to personal life, fix issues, set realistic priorities, spend time on realizing the happenings around, get life rolling smooth and then come back to work. This helps sustain in the longer run. Otherwise, with no self, everything can fall apart catastrophically.

From that phase, I learned many things. I now have much more respect for people who really have been successful at committing a large amount of their time on a volunteer project like Debian. Having myself gone through the time crunch phase, I can only imagine how many of the fellow DDs manage their time, sustainably, over the years. There are many folks I have seen active for more than a decade and they still rock.

1.8.2 release

Because the apt meta validation was a major issue, I have decided to run through the workflow and explain how apt-offline reacts to invalid tampered data. Below are konsole captures, with snipped output, where not very relevant.

rrs@priyasi:/var/tmp/Debian-Build/Result$ sudo apt-offline set /tmp/set.uris
Gathering details needed for 'update' operation
Gathering details needed for 'upgrade' operation
16:33 β™’ ΰ₯  ☺ πŸ˜„

One item to pay attention to, in this step, is some of the errors that are reported. Not all repository admins enable all the apt meta data available on their mirrors. This is commonly seen for localization related files. Similarly, not all compression types are available on all the repository servers. Some may only have .xz based meta files hosted while others may have .gz ones. So, for apt-offline, which has to bridge the gap of the offline <=> online setup, there is more work.

For compression types, apt-offline cycles through the known list of types. Only if, after cycling through all the known compression types, if the return is still a 404, then we error out.

Similarly, for localization related meta, we do the same cycling. But in addition to that, there is the possibility that the repository admin may not have enabled the localization data to be served at all. In that case, apt-offline ultimately will report and error.

And that is what is shown below. Because I see them not breaking the functionality, I treat them as non-fatal errors.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94

rrs@priyasi:/var/tmp/Debian-Build/Result$ apt-offline get /tmp/set-trimmed.uris --bundle /tmp/set.zip --threads 5

Fetching APT Data

WARNING: If you are on a slow connection, it is good to
WARNING: limit the number of threads to a low number like 2.
WARNING: Else higher number of threads executed could cause
WARNING: network congestion and timeouts.

Downloading http://deb.debian.org/debian/dists/testing/Release.gpg                                                             
Downloading http://deb.debian.org/debian/dists/testing/Release                                                             
Downloading http://deb.debian.org/debian/dists/testing/InRelease                                                             
Downloading http://deb.debian.org/debian/dists/unstable/Release.gpg                                                             
Downloading http://deb.debian.org/debian/dists/unstable/Release                                                             
http://deb.debian.org/debian/dists/unstable/Release.gpg done                                                             
Downloading http://deb.debian.org/debian/dists/unstable/InRelease                                                             
http://deb.debian.org/debian/dists/testing/Release.gpg done                                                             
Downloading http://deb.debian.org/debian/dists/experimental/Release.gpg                                                             
http://deb.debian.org/debian/dists/unstable/Release done                                                             
Downloading http://deb.debian.org/debian/dists/experimental/Release                                                             
http://deb.debian.org/debian/dists/testing/InRelease done                                                             
Downloading http://deb.debian.org/debian/dists/experimental/InRelease                                                             
http://deb.debian.org/debian/dists/testing/Release done                                                             
Downloading http://deb.debian.org/debian/dists/testing/main/source/Sources.xz                                                             
http://deb.debian.org/debian/dists/unstable/InRelease done                                                             
Downloading http://deb.debian.org/debian/dists/testing/non-free/source/Sources.xz                                                             
http://deb.debian.org/debian/dists/experimental/Release.gpg done                                                             
Downloading http://deb.debian.org/debian/dists/testing/contrib/source/Sources.xz                                                             
http://deb.debian.org/debian/dists/experimental/InRelease done                                                             
Downloading http://deb.debian.org/debian/dists/testing/main/binary-amd64/Packages.xz                                                             
http://deb.debian.org/debian/dists/experimental/Release done                                                             
Downloading http://deb.debian.org/debian/dists/testing/main/binary-i386/Packages.xz                                                             
http://deb.debian.org/debian/dists/testing/contrib/source/Sources.xz done                                                             
Downloading http://deb.debian.org/debian/dists/testing/main/binary-all/Packages.xz                                                             
http://deb.debian.org/debian/dists/testing/non-free/source/Sources.xz done                                                             
Downloading http://deb.debian.org/debian/dists/testing/main/i18n/Translation-en_IN.xz                                                             
ERROR: Giving up on URL http://deb.debian.org/debian/dists/testing/main/i18n/Translation-en_IN.lzma
Downloading http://deb.debian.org/debian/dists/testing/main/i18n/Translation-en.xz                                                             
http://deb.debian.org/debian/dists/testing/main/binary-all/Packages.xz done                                                             
Downloading http://deb.debian.org/debian/dists/testing/main/i18n/Translation-en_US.xz                                                             
http://deb.debian.org/debian/dists/testing/main/source/Sources.xz done                                                             
Downloading http://deb.debian.org/debian/dists/testing/main/Contents-amd64.xz                                                             
http://deb.debian.org/debian/dists/testing/main/i18n/Translation-en.bz2 done                                                             
ERROR: Giving up on URL http://deb.debian.org/debian/dists/testing/main/i18n/Translation-en_US.lzma
Downloading http://deb.debian.org/debian/dists/testing/main/Contents-i386.xz                                                             
Downloading http://deb.debian.org/debian/dists/testing/main/Contents-all.xz                                                             
ERROR: Giving up on URL http://deb.debian.org/debian/dists/testing/main/Contents-all.lzma
Downloading http://deb.debian.org/debian/dists/testing/non-free/binary-amd64/Packages.xz
http://deb.debian.org/debian/dists/testing/non-free/binary-amd64/Packages.xz done                                                             
Downloading http://deb.debian.org/debian/dists/testing/non-free/binary-i386/Packages.xz
http://deb.debian.org/debian/dists/testing/non-free/binary-i386/Packages.xz done                                                             
Downloading http://deb.debian.org/debian/dists/testing/non-free/binary-all/Packages.xz
http://deb.debian.org/debian/dists/testing/non-free/binary-all/Packages.xz done                                                             
Downloading http://deb.debian.org/debian/dists/testing/non-free/i18n/Translation-en_IN.xz
ERROR: Giving up on URL http://deb.debian.org/debian/dists/testing/non-free/i18n/Translation-en_IN.lzma
Downloading http://deb.debian.org/debian/dists/testing/non-free/i18n/Translation-en.xz
http://deb.debian.org/debian/dists/testing/non-free/i18n/Translation-en.bz2 done                                                             
Downloading http://deb.debian.org/debian/dists/testing/non-free/i18n/Translation-en_US.xz
http://deb.debian.org/debian/dists/testing/main/binary-i386/Packages.xz done                                                             
Downloading http://deb.debian.org/debian/dists/testing/non-free/Contents-amd64.xz                                                             
ERROR: Giving up on URL http://deb.debian.org/debian/dists/testing/non-free/i18n/Translation-en_US.lzma
Downloading http://deb.debian.org/debian/dists/testing/non-free/Contents-i386.xz                                                             
http://deb.debian.org/debian/dists/testing/non-free/Contents-i386.gz done                                                             
Downloading http://deb.debian.org/debian/dists/testing/non-free/Contents-all.xz                                                             
http://deb.debian.org/debian/dists/testing/non-free/Contents-amd64.gz done                                                             
http://deb.debian.org/debian/dists/testing/main/binary-amd64/Packages.xz done                                                             
Downloading http://deb.debian.org/debian/dists/testing/contrib/binary-amd64/Packages.xz
Downloading http://deb.debian.org/debian/dists/testing/contrib/binary-i386/Packages.xz
http://deb.debian.org/debian/dists/testing/contrib/binary-amd64/Packages.xz done                                                             
Downloading http://deb.debian.org/debian/dists/testing/contrib/binary-all/Packages.xz                                                             
http://deb.debian.org/debian/dists/testing/contrib/binary-i386/Packages.xz done                                                             
http://deb.debian.org/debian/dists/testing/contrib/binary-all/Packages.xz done                                                             
Downloading http://deb.debian.org/debian/dists/testing/contrib/i18n/Translation-en_IN.xz
Downloading http://deb.debian.org/debian/dists/testing/contrib/i18n/Translation-en.xz                                                             
ERROR: Giving up on URL http://deb.debian.org/debian/dists/testing/non-free/Contents-all.lzma
Downloading http://deb.debian.org/debian/dists/testing/contrib/i18n/Translation-en_US.xz
http://deb.debian.org/debian/dists/testing/contrib/i18n/Translation-en.bz2 done                                                             
Downloading http://deb.debian.org/debian/dists/testing/contrib/Contents-amd64.xz                                                             
ERROR: Giving up on URL http://deb.debian.org/debian/dists/testing/contrib/i18n/Translation-en_IN.lzma
Downloading http://deb.debian.org/debian/dists/testing/contrib/Contents-i386.xz                                                             
ERROR: Giving up on URL http://deb.debian.org/debian/dists/testing/contrib/i18n/Translation-en_US.lzma
Downloading http://deb.debian.org/debian/dists/testing/contrib/Contents-all.xz                                                             
http://deb.debian.org/debian/dists/testing/contrib/Contents-amd64.gz done                                                             
http://deb.debian.org/debian/dists/testing/contrib/Contents-i386.gz done                                                             
ERROR: Giving up on URL http://deb.debian.org/debian/dists/testing/contrib/Contents-all.lzma
http://deb.debian.org/debian/dists/testing/main/Contents-i386.gz done                                                             
http://deb.debian.org/debian/dists/testing/main/Contents-amd64.gz done                                                             
 81 /  81 items: [##############################] 100.0% of 101 MiB
Downloaded data to /tmp/set.zip
ERROR: Some items failed to download. Downloaded data may be incomplete
ERROR: Please run in verbose mode to see details about failed items


16:38 β™’ ΰ₯   ☹ 😟=> 100

First, lets unpack the archive file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

rrs@priyasi:/var/tmp/Debian-Build/Result$ cd  /tmp/
16:39 β™’ ΰ₯  ☺ πŸ˜„    
rrs@priyasi:/tmp$ mkdir set-folder
16:39 β™’ ΰ₯  ☺ πŸ˜„    
rrs@priyasi:/tmp$ cd set-folder/
16:39 β™’ ΰ₯  ☺ πŸ˜„    
rrs@priyasi:/tmp/set-folder$ unzip ../set.zip 
Archive:  ../set.zip
  inflating: deb.debian.org_debian_dists_unstable_Release.gpg  
  inflating: deb.debian.org_debian_dists_testing_Release.gpg  
  inflating: deb.debian.org_debian_dists_unstable_Release  
  inflating: deb.debian.org_debian_dists_testing_InRelease  
  inflating: deb.debian.org_debian_dists_testing_Release  
  inflating: deb.debian.org_debian_dists_unstable_InRelease  
  inflating: deb.debian.org_debian_dists_experimental_Release.gpg  
  inflating: deb.debian.org_debian_dists_experimental_InRelease  
  inflating: deb.debian.org_debian_dists_experimental_Release  
  inflating: deb.debian.org_debian_dists_testing_contrib_source_Sources.xz  
  inflating: deb.debian.org_debian_dists_testing_non-free_source_Sources.xz  
  inflating: deb.debian.org_debian_dists_testing_main_binary-all_Packages.xz  
  inflating: deb.debian.org_debian_dists_testing_main_source_Sources.xz  
  inflating: deb.debian.org_debian_dists_testing_main_i18n_Translation-en.bz2  
  inflating: deb.debian.org_debian_dists_testing_non-free_binary-amd64_Packages.xz  
  inflating: deb.debian.org_debian_dists_testing_non-free_binary-i386_Packages.xz  
  inflating: deb.debian.org_debian_dists_testing_non-free_binary-all_Packages.xz  
  inflating: deb.debian.org_debian_dists_testing_non-free_i18n_Translation-en.bz2  
  inflating: deb.debian.org_debian_dists_testing_main_binary-i386_Packages.xz  
  inflating: deb.debian.org_debian_dists_testing_non-free_Contents-i386.gz  
  inflating: deb.debian.org_debian_dists_testing_non-free_Contents-amd64.gz  
  inflating: deb.debian.org_debian_dists_testing_main_binary-amd64_Packages.xz  
  inflating: deb.debian.org_debian_dists_testing_contrib_binary-amd64_Packages.xz  
  inflating: deb.debian.org_debian_dists_testing_contrib_binary-i386_Packages.xz  
  inflating: deb.debian.org_debian_dists_testing_contrib_binary-all_Packages.xz  
  inflating: deb.debian.org_debian_dists_testing_contrib_i18n_Translation-en.bz2  
  inflating: deb.debian.org_debian_dists_testing_contrib_Contents-amd64.gz  
  inflating: deb.debian.org_debian_dists_testing_contrib_Contents-i386.gz  
  inflating: deb.debian.org_debian_dists_testing_main_Contents-i386.gz  
  inflating: deb.debian.org_debian_dists_testing_main_Contents-amd64.gz  
16:39 β™’ ΰ₯  ☺ πŸ˜„    
rrs@priyasi:/tmp/set-folder$ ls
deb.debian.org_debian_dists_experimental_InRelease                    deb.debian.org_debian_dists_testing_main_Contents-i386.gz
deb.debian.org_debian_dists_experimental_Release                      deb.debian.org_debian_dists_testing_main_i18n_Translation-en.bz2
deb.debian.org_debian_dists_experimental_Release.gpg                  deb.debian.org_debian_dists_testing_main_source_Sources.xz
deb.debian.org_debian_dists_testing_contrib_binary-all_Packages.xz    deb.debian.org_debian_dists_testing_non-free_binary-all_Packages.xz
deb.debian.org_debian_dists_testing_contrib_binary-amd64_Packages.xz  deb.debian.org_debian_dists_testing_non-free_binary-amd64_Packages.xz
deb.debian.org_debian_dists_testing_contrib_binary-i386_Packages.xz   deb.debian.org_debian_dists_testing_non-free_binary-i386_Packages.xz
deb.debian.org_debian_dists_testing_contrib_Contents-amd64.gz         deb.debian.org_debian_dists_testing_non-free_Contents-amd64.gz
deb.debian.org_debian_dists_testing_contrib_Contents-i386.gz          deb.debian.org_debian_dists_testing_non-free_Contents-i386.gz
deb.debian.org_debian_dists_testing_contrib_i18n_Translation-en.bz2   deb.debian.org_debian_dists_testing_non-free_i18n_Translation-en.bz2
deb.debian.org_debian_dists_testing_contrib_source_Sources.xz         deb.debian.org_debian_dists_testing_non-free_source_Sources.xz
deb.debian.org_debian_dists_testing_InRelease                         deb.debian.org_debian_dists_testing_Release
deb.debian.org_debian_dists_testing_main_binary-all_Packages.xz       deb.debian.org_debian_dists_testing_Release.gpg
deb.debian.org_debian_dists_testing_main_binary-amd64_Packages.xz     deb.debian.org_debian_dists_unstable_InRelease
deb.debian.org_debian_dists_testing_main_binary-i386_Packages.xz      deb.debian.org_debian_dists_unstable_Release
deb.debian.org_debian_dists_testing_main_Contents-amd64.gz            deb.debian.org_debian_dists_unstable_Release.gpg
16:39 β™’ ΰ₯  ☺ πŸ˜„    
1
2

rrs@priyasi:/tmp/set-folder$ echo 112312312321 >> deb.debian.org_debian_dists_testing_non-free_source_Sources.xz
16:40 β™’ ΰ₯  ☺ πŸ˜„

So in this step, we tell apt-offline to install the downloaded files. This will also include the tampered file. The output you see below is standard and reports everything to have succeeded.

But note that the tampered file is not in the list of synced files. That file is just simply missing from the list.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

rrs@priyasi:/tmp/set-folder$ sudo apt-offline install .
Proceeding with installation
gpgv: Signature made Friday 07 February 2020 01:55:24 PM IST
gpgv:                using RSA key 16E90B3FDF65EDE3AA7F323C04EE7237B7D453EC
gpgv: Good signature from "Debian Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>"
gpgv: Signature made Friday 07 February 2020 01:55:43 PM IST
gpgv:                using RSA key 0146DC6D4A0B2914BDED34DB648ACFD622F3D138
gpgv: Good signature from "Debian Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>"
gpgv: Signature made Friday 07 February 2020 01:56:44 PM IST
gpgv:                using RSA key 16E90B3FDF65EDE3AA7F323C04EE7237B7D453EC
gpgv: Good signature from "Debian Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>"
gpgv: Signature made Friday 07 February 2020 01:56:45 PM IST
gpgv:                using RSA key 0146DC6D4A0B2914BDED34DB648ACFD622F3D138
gpgv: Good signature from "Debian Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>"
gpgv: Signature made Friday 07 February 2020 01:56:58 PM IST
gpgv:                using RSA key 16E90B3FDF65EDE3AA7F323C04EE7237B7D453EC
gpgv: Good signature from "Debian Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>"
gpgv: Signature made Friday 07 February 2020 01:56:58 PM IST
gpgv:                using RSA key 0146DC6D4A0B2914BDED34DB648ACFD622F3D138
gpgv: Good signature from "Debian Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>"
deb.debian.org_debian_dists_testing_contrib_Contents-amd64.gz synced.
deb.debian.org_debian_dists_testing_contrib_Contents-i386.gz synced.
deb.debian.org_debian_dists_testing_contrib_binary-all_Packages.xz synced.
deb.debian.org_debian_dists_testing_contrib_binary-amd64_Packages.xz synced.
deb.debian.org_debian_dists_testing_contrib_binary-i386_Packages.xz synced.
deb.debian.org_debian_dists_testing_contrib_i18n_Translation-en.bz2 synced.
deb.debian.org_debian_dists_testing_contrib_source_Sources.xz synced.
deb.debian.org_debian_dists_testing_main_Contents-amd64.gz synced.
deb.debian.org_debian_dists_testing_main_Contents-i386.gz synced.
deb.debian.org_debian_dists_testing_main_binary-all_Packages.xz synced.
deb.debian.org_debian_dists_testing_main_binary-amd64_Packages.xz synced.
deb.debian.org_debian_dists_testing_main_binary-i386_Packages.xz synced.
deb.debian.org_debian_dists_testing_main_i18n_Translation-en.bz2 synced.
deb.debian.org_debian_dists_testing_main_source_Sources.xz synced.
deb.debian.org_debian_dists_testing_non-free_Contents-amd64.gz synced.
deb.debian.org_debian_dists_testing_non-free_Contents-i386.gz synced.
deb.debian.org_debian_dists_testing_non-free_binary-all_Packages.xz synced.
deb.debian.org_debian_dists_testing_non-free_binary-amd64_Packages.xz synced.
deb.debian.org_debian_dists_testing_non-free_binary-i386_Packages.xz synced.
deb.debian.org_debian_dists_testing_non-free_i18n_Translation-en.bz2 synced.
16:41 β™’ ΰ₯  ☺ πŸ˜„

So, in the above example, apt-offline discarded the tampered file and the final exit of the command was a success. Now, let’s run the same command with the ‘–verbose’ switch. Below is the output.

Notice the highlighted line below, where it reports that the file is tampered and does not match the checksum

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

rrs@priyasi:/tmp/set-folder$ sudo apt-offline install . --verbose
VERBOSE: Namespace(allow_unauthenticated=False, func=<function installer at 0x7f6a6c7c54d0>, install='.', install_simulate=False, install_src_path=None, skip_bug_reports=False, skip_changelog=False, strict_deb_check=False, verbose=True)
VERBOSE: No changelog available
Proceeding with installation
VERBOSE: {}
VERBOSE: Great!!! No bugs found for all the packages that were downloaded.

VERBOSE: APT Signature verification path is: ['/etc/apt/trusted.gpg.d/', '/etc/apt/trusted.gpg']
VERBOSE: Adding /etc/apt/trusted.gpg.d/debian-archive-jessie-automatic.gpg to the apt-offline keyring
VERBOSE: Adding /etc/apt/trusted.gpg.d/debian-archive-jessie-security-automatic.gpg to the apt-offline keyring
VERBOSE: Adding /etc/apt/trusted.gpg.d/debian-archive-jessie-stable.gpg to the apt-offline keyring
VERBOSE: Adding /etc/apt/trusted.gpg.d/debian-archive-stretch-automatic.gpg to the apt-offline keyring
VERBOSE: Adding /etc/apt/trusted.gpg.d/debian-archive-stretch-security-automatic.gpg to the apt-offline keyring
VERBOSE: Adding /etc/apt/trusted.gpg.d/debian-archive-stretch-stable.gpg to the apt-offline keyring
VERBOSE: Adding /etc/apt/trusted.gpg.d/debian-archive-buster-automatic.gpg to the apt-offline keyring
VERBOSE: Adding /etc/apt/trusted.gpg.d/debian-archive-buster-security-automatic.gpg to the apt-offline keyring

.....snipped.....

VERBOSE: localFile ./deb.debian.org_debian_dists_testing_non-free_Contents-amd64.gz Integrity with checksum 024957d30be2acbb9e66c9802f825115d32437420300a2b28ab60ae4ecb76fcf matches
VERBOSE: localFile ./deb.debian.org_debian_dists_testing_non-free_Contents-i386.gz Integrity with checksum 5266d2f3ea41c4e988e71b4bbe58dd1178a23ce1ed50908c73a0cb39201136e3 matches
VERBOSE: localFile ./deb.debian.org_debian_dists_testing_non-free_binary-all_Packages.xz Integrity with checksum 9f0f3aa5560452d45f82c5121ea844c68e641c8fbb56ef69d570c641b6cce662 matches
VERBOSE: localFile ./deb.debian.org_debian_dists_testing_non-free_binary-amd64_Packages.xz Integrity with checksum 811f7752a13dfcbd748478dda267fb810c52fc14769d2d5c7871c75e35350d66 matches
VERBOSE: localFile ./deb.debian.org_debian_dists_testing_non-free_binary-i386_Packages.xz Integrity with checksum 7df3512b5da7258613774921023d68c71858d89fddafd694e2dfd19cef54314b matches
VERBOSE: localFile ./deb.debian.org_debian_dists_testing_non-free_i18n_Translation-en.bz2 Integrity with checksum 1bf3cd0cff6fadf1a74280912c3229374344cd6c347d2f533b001843d84b236d matches
VERBOSE: localFile ./deb.debian.org_debian_dists_testing_non-free_source_Sources.xz integrity doesn't match to checksum a94589ab3c204bb4d710d72ea21abac8007b14e5c5dacbe43be07c51ba5f0a0a
VERBOSE: Synchronized file to /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_Contents-amd64
VERBOSE: /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_Contents-amd64 file synced to APT.
deb.debian.org_debian_dists_testing_contrib_Contents-amd64.gz synced.
VERBOSE: Synchronized file to /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_Contents-i386
VERBOSE: /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_Contents-i386 file synced to APT.
deb.debian.org_debian_dists_testing_contrib_Contents-i386.gz synced.
VERBOSE: Synchronized file to /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_binary-all_Packages
VERBOSE: /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_binary-all_Packages file synced to APT.
deb.debian.org_debian_dists_testing_contrib_binary-all_Packages.xz synced.
VERBOSE: Synchronized file to /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_binary-amd64_Packages
VERBOSE: /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_binary-amd64_Packages file synced to APT.
deb.debian.org_debian_dists_testing_contrib_binary-amd64_Packages.xz synced.
VERBOSE: Synchronized file to /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_binary-i386_Packages
VERBOSE: /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_binary-i386_Packages file synced to APT.
deb.debian.org_debian_dists_testing_contrib_binary-i386_Packages.xz synced.

.....snipped.....

16:42 β™’ ΰ₯  ☺ πŸ˜„

This is pretty much the validation required and done by apt-offline for apt meta Packages files.

Please do file bug reports if you think the overall exit status of apt-offline under such scenarios should be different than what it is currently.

For the tampered meta Packages files:

  • should the visual representation be different ?
  • Should an error be printed ?
  • What about the exit status ?

Similarly, for the ‘get’ operation:

  • Should we do something different for non-existing localization files on the repository server ?
  • Is there any different way to go through the supported list of compression types for meta files ?

apt-offline allows a user to install a new package and all its dependencies easily on the offline machine. The below workflow will demonstrate the same and will also go through the tampering of the .deb files and see how apt-offline/apt deals with it.

In below example, a user wants to install the gnome-todo package on the offline machine, which has a couple of dependencies.

rrs@priyasi:/tmp/set-folder$ sudo apt install gnome-todo
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
The following NEW packages will be installed:
  gnome-todo gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
0 upgraded, 5 newly installed, 0 to remove and 1 not upgraded.
Need to get 784 kB of archives.
After this operation, 2,337 kB of additional disk space will be used.
Do you want to continue? [Y/n] n
Abort.
16:47 β™’ ΰ₯   ☹ 😟=> 1

The below command generates a (signature) file, which will include all details about requested package and its dependencies.

1
2
3

rrs@priyasi:/tmp/set-folder$ sudo apt-offline set /tmp/gnome-todo.uris --install-packages gnome-todo
Gathering installation details for package ['gnome-todo']
16:48 β™’ ΰ₯  ☺ πŸ˜„

Below is the usual step to be performed on the online machine with the generated gnome-todo.uris signature file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

rrs@priyasi:/tmp/set-folder$ apt-offline get /tmp/gnome-todo.uris --download-dir /tmp/gnome-todo --bug-reports --threads 3

Fetching APT Data

WARNING: If you are on a slow connection, it is good to
WARNING: limit the number of threads to a low number like 2.
WARNING: Else higher number of threads executed could cause
WARNING: network congestion and timeouts.

Downloading libpeas-common - 187 KiB                                                             
Downloading libpeas-1.0-0 - 196 KiB                                                             
Downloading gnome-todo-common - 228 KiB                                                             
libpeas-common done                                                             
Fetching bug report for libpeas-common                                                            
libpeas-1.0-0 done                                                             
Fetching bug report for libpeas-1.0-0                                                            
gnome-todo-common done                                                             
Fetching bug report for gnome-todo-common                                                            
Fetched bug report for libpeas-common                                                            
Downloading libgnome-todo - 6 KiB                                                             
libgnome-todo done                                                             
Fetching bug report for libgnome-todo                                                            
Fetched bug report for gnome-todo-common                                                            
Downloading gnome-todo - 146 KiB                                                             
gnome-todo done                                                             
Fetching bug report for gnome-todo                                                            
Fetched bug report for libpeas-1.0-0                                                            
Fetched bug report for libgnome-todo                                                            
Fetched bug report for gnome-todo                                                            
  5 /   5 items: [##############################] 100.0% of 765 KiB
Downloaded data to /tmp/gnome-todo
16:49 β™’ ΰ₯  ☺ πŸ˜„

Note: The fact is that apt-offline will not do any checksum validation for the .deb files. The validation is completely delegated to apt.

rrs@priyasi:/tmp/gnome-todo$ sudo apt-offline install -h
usage: apt-offline install [-h] [--verbose] [--simulate]
                           [--install-src-path INSTALL_SRC_PATH]
                           [--skip-bug-reports] [--skip-changelog]
                           [--allow-unauthenticated] [--strict-deb-check]
                           apt-offline-download.zip | apt-offline-download/

positional arguments:
  apt-offline-download.zip | apt-offline-download/
                        Install apt-offline data, a bundle file or a directory

optional arguments:
  -h, --help            show this help message and exit
  --verbose             Enable verbose messages
  --simulate            Just simulate. Very helpful when debugging
  --install-src-path INSTALL_SRC_PATH
                        Install src packages to specified path.
  --skip-bug-reports    Skip the bug report check
  --skip-changelog      Skip display of changelog
  --allow-unauthenticated
                        Ignore apt gpg signatures mismatch
  --strict-deb-check    Perform strict checksum validaton for downloaded .deb
                        files
16:50 β™’ ΰ₯  ☺ πŸ˜„

and from the manpage:

       --strict-deb-check
                 With  this option enabled, apt-offline delegate's .deb package checksum validation to apt. While the .debs are already avail‐
                 able, they are stored in the temporary apt cache, where apt validates its checksum, before considering it  for  further  pro‐
                 cessing.   Note:  This  does  have the caveat that apt may need network availability even though it doesn't download anything
                 over the network. But it does invoke the download routines and realizes that the payload is already available. It  then  fur‐
                 ther proceeds with checksum validation

                 The  default  behavior  is to not do strict checksum validation for .deb files. Instead, apt-offline copies the .deb files to
                 apt's download location. apt still does size validation of the available .deb files and discards them in case there is a mis‐
                 match.

Before we proceed with the example of checksum verification for .deb files, lets do a pristine run of the downloaded files, without any tampering to them.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

rrs@priyasi:/tmp/gnome-todo$ sudo apt-offline install .
Proceeding with installation


Following are the list of bugs present.
822525  gnome-todo      : gnome-todo: Memory leak while loading local and remote lists
853114  gnome-todo      : no longer loads caldav lists
883961  libgnome-todo   : libgnome-todo: Not actually a library
829470  libpeas-1.0-0   : libpeas: Python Plugin Broken
(Y) Yes. Proceed with installation
(N) No, Abort.
(R) Redisplay the list of bugs.
(Bug Number) Display the bug report from the Offline Bug Reports.
(?) Display this help message.
What would you like to do next:  (y, N, ?)y
gnome-todo_3.28.1-5_amd64.deb file synced.
libgnome-todo_3.28.1-5_amd64.deb file synced.
gnome-todo-common_3.28.1-5_all.deb file synced.
libpeas-1.0-0_1.22.0-5_amd64.deb file synced.
libpeas-common_1.22.0-5_all.deb file synced.
16:51 β™’ ΰ₯  ☺ πŸ˜„    
rrs@priyasi:/tmp/gnome-todo$ sudo apt install gnome-todo
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
The following NEW packages will be installed:
  gnome-todo gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
0 upgraded, 5 newly installed, 0 to remove and 1 not upgraded.
Need to get 0 B/784 kB of archives.
After this operation, 2,337 kB of additional disk space will be used.
Do you want to continue? [Y/n] n
Abort.
16:51 β™’ ΰ₯   ☹ 😟=> 1

In the above example, everything is clean and all requirements to apt are satisfied.

Here’s one more exaple, where we invoke the non-default --strict-deb-check option.

Everything remains the same, but apt gives a prompt saying that it needs to download the payload from the web. The reality is that if you just proceed with yes, nothing gets downloaded.

Note: It is not possible to explain that with a still presentation and I’m lazy to make a motion object of it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

rrs@priyasi:/tmp/gnome-todo$ sudo apt-offline install . --strict-deb-check 
Proceeding with installation


Following are the list of bugs present.
822525  gnome-todo      : gnome-todo: Memory leak while loading local and remote lists
853114  gnome-todo      : no longer loads caldav lists
883961  libgnome-todo   : libgnome-todo: Not actually a library
829470  libpeas-1.0-0   : libpeas: Python Plugin Broken
(Y) Yes. Proceed with installation
(N) No, Abort.
(R) Redisplay the list of bugs.
(Bug Number) Display the bug report from the Offline Bug Reports.
(?) Display this help message.
What would you like to do next:  (y, N, ?)y
gnome-todo_3.28.1-5_amd64.deb file synced.
libgnome-todo_3.28.1-5_amd64.deb file synced.
gnome-todo-common_3.28.1-5_all.deb file synced.
libpeas-1.0-0_1.22.0-5_amd64.deb file synced.
libpeas-common_1.22.0-5_all.deb file synced.
16:52 β™’ ΰ₯  ☺ πŸ˜„    
rrs@priyasi:/tmp/gnome-todo$ sudo apt install gnome-todo
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
The following NEW packages will be installed:
  gnome-todo gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
0 upgraded, 5 newly installed, 0 to remove and 1 not upgraded.
Need to get 784 kB of archives.
After this operation, 2,337 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://deb.debian.org/debian testing/main amd64 libpeas-common all 1.22.0-5 [192 kB]
Get:2 http://deb.debian.org/debian testing/main amd64 libpeas-1.0-0 amd64 1.22.0-5 [201 kB]
Get:3 http://deb.debian.org/debian testing/main amd64 gnome-todo-common all 3.28.1-5 [234 kB]
Get:4 http://deb.debian.org/debian testing/main amd64 libgnome-todo amd64 3.28.1-5 [6,260 B]
Get:5 http://deb.debian.org/debian testing/main amd64 gnome-todo amd64 3.28.1-5 [150 kB]
Retrieving bug reports... Done
Parsing Found/Fixed information... Done
.....snipped.....
16:53 β™’ ΰ₯  ☺ πŸ˜„

To sum it up, this one is an odd case because though nothing for the debs is downloaded, BUT, the network needs to be active for this co-routine to run. If, say, the network is unavailable, apt complains. I haven’t checked, but apt does invoke some network code.

But no payload is downloaded. apt just validates and realizes that all the to-be-downloaded data, is intact and available.

rrs@priyasi:/tmp/gnome-todo$ echo fasdfadsfasdfasdfasd >> gnome-todo_3.28.1-5_amd64.deb
16:54 β™’ ΰ₯  ☺ πŸ˜„    

rrs@priyasi:/tmp/gnome-todo$ sudo apt clean
16:54 β™’ ΰ₯  ☺ πŸ˜„

So we tampered one of the .deb files, gnome-todo_3.28.1-5_amd64.deb. And ask apt-offline to run its ‘install’ operation along with the new --strict-deb-check option.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

rrs@priyasi:/tmp/gnome-todo$ sudo apt-offline install . --strict-deb-check 
Proceeding with installation


Following are the list of bugs present.
822525  gnome-todo      : gnome-todo: Memory leak while loading local and remote lists
853114  gnome-todo      : no longer loads caldav lists
883961  libgnome-todo   : libgnome-todo: Not actually a library
829470  libpeas-1.0-0   : libpeas: Python Plugin Broken
(Y) Yes. Proceed with installation
(N) No, Abort.
(R) Redisplay the list of bugs.
(Bug Number) Display the bug report from the Offline Bug Reports.
(?) Display this help message.
What would you like to do next:  (y, N, ?)y
gnome-todo_3.28.1-5_amd64.deb file synced.
libgnome-todo_3.28.1-5_amd64.deb file synced.
gnome-todo-common_3.28.1-5_all.deb file synced.
libpeas-1.0-0_1.22.0-5_amd64.deb file synced.
libpeas-common_1.22.0-5_all.deb file synced.
16:54 β™’ ΰ₯  ☺ πŸ˜„    

rrs@priyasi:/tmp/gnome-todo$ sudo apt install gnome-todo
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
The following NEW packages will be installed:
  gnome-todo gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
0 upgraded, 5 newly installed, 0 to remove and 1 not upgraded.
Need to get 784 kB of archives.
After this operation, 2,337 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://deb.debian.org/debian testing/main amd64 libpeas-common all 1.22.0-5 [192 kB]
Get:2 http://deb.debian.org/debian testing/main amd64 libpeas-1.0-0 amd64 1.22.0-5 [201 kB]
Get:3 http://deb.debian.org/debian testing/main amd64 gnome-todo-common all 3.28.1-5 [234 kB]
Get:4 http://deb.debian.org/debian testing/main amd64 libgnome-todo amd64 3.28.1-5 [6,260 B]
Get:5 http://deb.debian.org/debian testing/main amd64 gnome-todo amd64 3.28.1-5 [150 kB]
Fetched 150 kB in 1s (141 kB/s)     
Retrieving bug reports... Done
Parsing Found/Fixed information... Done

16:55 β™’ ΰ₯   ☹ 😟=> 100

Pay attention to the downloaded data which is only 150 KiB, for the gnome-todo package, which was tampered. Even though apt stated that it needs to download 784 KiB of data, it actually downloaded 150 KiB only. All data was already downloaded by apt-offline but we had tampered one of the files, which resulted in it being re-downloaded.

Now, lets do one more run with the default behavior of apt-offline, i.e. without the --strict-deb-check option. This will result in apt (internally) detecting the tampering and prompting the user that the (tampered) file needs to be downloaded

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

rrs@priyasi:/tmp/gnome-todo$ sudo apt-offline install .
Proceeding with installation


Following are the list of bugs present.
822525  gnome-todo      : gnome-todo: Memory leak while loading local and remote lists
853114  gnome-todo      : no longer loads caldav lists
883961  libgnome-todo   : libgnome-todo: Not actually a library
829470  libpeas-1.0-0   : libpeas: Python Plugin Broken
(Y) Yes. Proceed with installation
(N) No, Abort.
(R) Redisplay the list of bugs.
(Bug Number) Display the bug report from the Offline Bug Reports.
(?) Display this help message.
What would you like to do next:  (y, N, ?)y
gnome-todo_3.28.1-5_amd64.deb file synced.
libgnome-todo_3.28.1-5_amd64.deb file synced.
gnome-todo-common_3.28.1-5_all.deb file synced.
libpeas-1.0-0_1.22.0-5_amd64.deb file synced.
libpeas-common_1.22.0-5_all.deb file synced.
16:56 β™’ ΰ₯  ☺ πŸ˜„    
rrs@priyasi:/tmp/gnome-todo$ sudo apt^C
16:56 β™’ ΰ₯   ☹ 😟=> 130  
rrs@priyasi:/tmp/gnome-todo$ sudo apt install gnome-todo
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
The following NEW packages will be installed:
  gnome-todo gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
0 upgraded, 5 newly installed, 0 to remove and 1 not upgraded.
Need to get 150 kB/784 kB of archives.
After this operation, 2,337 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://deb.debian.org/debian testing/main amd64 gnome-todo amd64 3.28.1-5 [150 kB]
Fetched 150 kB in 0s (448 kB/s)     
Retrieving bug reports... Done
Parsing Found/Fixed information... Done
.....snipped......
16:57 β™’ ΰ₯  ☺ πŸ˜„

Notice the highlighted line, which gives a less confusing, realistic summary of what needs to be done. In this case, apt is prompting the user that 150 KiB of data needs to be downloaded, which indeed is the case.

Linux  Debian  APT  apt-offline  Package Management 

See also