By default gzip runs at level 6. The reason tar takes so little time compared to gzip is that there's very little computational overhead in copying your files into a single file which is what it does. Enter pigz , which can use multiple threads to perform the compression. An example of how to use this would be:. There is a nice succint summary of the --use-compress-program option over on a sister site.
If you are stuck to using just gzip for the time being and have multiple files to compress, you could try compressing them individually - that way you'll use more of that multi-core CPU by running more than one process in parallel. One can exploit the number of process available as well in pigz which is usually faster performance as shown in the following command. This is probably faster than the methods suggested in the post as -p is the number of processes one can run. In my personal experience setting a very large value doesnt hurt performance if the directory to be archived consists of a large number of small files.
Else the default value considered is 8. For large files, my recommendation would be to set this value as the total number of threads supported on the system. Default value is 6 for compression.
Sign up to join this community. The best answers are voted up and rise to the top. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Learn more. Time to zip very large G files Ask Question. Asked 8 years, 6 months ago. Active 2 years, 5 months ago. Some proxies might cache gzipped resources without also caching their Content-Encoding, or even try to re-compress compressed content. The Vary HTTP response header specifies how proxies and caches handle compressed content and should be enabled whether dynamic or static compression is enabled.
Before you can use static compression, you will need to create a gzipped copy of each file you want to serve. To load the plugin, add the following line to your Apache configuration file:. As with Nginx, you can enable gzip for the entire web server or for a specific configuration block.
Apache also supports compression for certain file types, setting compression levels, and managing proxy settings. This example uses this method to serve pre-compressed CSS files:. To show the difference between compressed and uncompressed websites, we ran a page speed test on a website with three different configurations: one with compression enabled, one with compression completely disabled, and one serving only pre-compressed content.
We created a basic website using Hugo and hosted it on an f1-micro Google Compute Engine instance running Nginx version 1. For the gzip-enabled versions, we used the default settings for both Nginx and the gzip command-line application. To run the test, we used a recurring page speed check to contact the site every 30 minutes. After four runs, we reconfigured and restarted the Nginx server for the next test. We dropped the first run to allow time for the Nginx server to warm up.
We verified that Nginx was using static compression and not dynamic compression by using strace to see which files were being accessed:. With dynamic compression, the web server transferred For some files, such as bootstrap. Although this was a small site with few optimizations, simply enabling gzip on the web server allowed for a significant savings in load time. The fact that static compression performed roughly the same as dynamic compression also shows that for smaller sites, the cost of dynamic compression on system resources is minor.
Websites with larger files and higher traffic volumes will likely see a more significant amount of CPU usage, and would benefit more from static compression. It tells you how it's compressed and how to decompress it if it knows how. OpenBSD's tar doesn't support xz so it can't help there, but does support gzip so it suggests using -z. Not letting untrusted input automatically increase the attack surface it's exposed to is a feature.
How is that a feature? The user's explicitly asking for this. This feature reminds me of vim, that suggests closing with ":quit" when you press C-x C-c i. It knows full well what you want to do and even has special code to handle it, but then insists to hand you more work. Vim suggests closing with ":quit" when you hit C-c; the C-x is irrelevant. Upon receiving a C-c, it does not know full well what the user wants to do.
When vim receives a C-c from you or someone who just stumbled into vim and doesn't know how to exit the user wants to exit. When vim receives a C-c from me , it's because I meant to kill the process I spawned from vim, and it ended before the key was pressed.
I very much do not want it to quit on me at that point. Showing a message seems the best compromise. I don't really care what vim does, that's a different argument. There have been many vulnerabilities in gzip, and in tar implementations that let untrusted input choose how it gets parsed, those vulnerabilities might as well be in tar itself. The same applies to OGG audio. Better than MP3 but the average user is unable to play it.
So everybody just sticks to MP3. MP3s with a decent bit rate is as good is it gets. DanBC on Dec 12, root parent next [—]. But MP3 was patent encumbered and so a bunch of music creation software had weird work-arounds. I am not sure what your point is? Yes MP3 files were not the right choice years ago, but today who cares. If I recall correctly, there are some alive patents in the U. The workaround that the parent is talking about is usually "get LAME from a different distributor", which is still done by Audacity and others.
That's the thing. With better formats, you don't need kbps for transparency. For those of us that sample from songs that we buy, WAVs are a bit easier to work with because the DAW doesn't have to spend time converting it.
That said, since most of my tracks these days are using either the 48k or 96k sample rate, it still needs to be converted from But can you tell the difference between a variable encoded Kbps MP3 using a modern encoder and a wav file? I have some reasonable equipment and I most definantly can't.
On performance equipment I could tell VBR was not holding up. I don't think I ever heard the difference. I have to say I have never tried slowing down the music to try and hear the difference so it is possible under these conditions that it might make a difference.
I would say yes, but again, playing music at very amplified volumes and tweaking its tempo is not a common use-case. They say they can, but under scrutiny it generally falls apart.
At some point, you're listening to sound and not music anyway. The degree of difference depends on the kind of music you listen to. Live recordings of acoustic ensembles in airy cathedrals -- in that case you can tell the difference.
On tracks that have a highly produced studio sound, where everything is an electronic instrument -- not going to be much of a difference. I tried doing tests like this and I could not find any recordings where I could tell the difference at kbps VBR. I not saying that it is impossible, but the conditions must be pretty rare and the difference very minor - compared to the massive degradation that come from room effects it amounts to nothing. Chamber music in an echo-y cathedral.
With bad encoding, you can hear a noticeable difference in the length of time the reverberations are audible and and the timbre of those reverberations cab be quite different.
With lots of acoustic music, the "accidental beauty" produced by such effects can be quite important. Finding this convinced me to re-encode my music collection in kbps MP3 for anything high quality, and algorithmically chosen variable bitrates for lower quality recordings -- usually around kbps. That was quite a number of years ago, though. I'd probably use another format today. That's not true. MP3 simply never gets transparent and you can notice with 5 dollar in-ears.
And people in general notice. This leads to absurdities such as bitrates of kbps, even thou these do not sound significantly better than kbps and are still not transparent. On the other hand kbps AAC is transparent for almost any input. AAC is supported abou everywhere where mp3 is. The quality alone should be convincing. The smaller size make the usage of mp3 IMHO insane. I have listened to a lot of MP3 at different bit rates and with modern encoders and variable bit rates I can't tell the difference between anything above Kbps - most of the time it is hard to tell the difference between kbp and anything higher.
Really at kps you are entering the realm of fantasy if you think you can hear any difference. I absolutely heard a difference between and everything below. You can tell me I didn't, but I did. There is a world of difference between Kbps and , and is a lot worse.
If you can't hear it, I understand, but the blame isn't the algorithm -- it is your equipment, your song selection, or your ears. This is not true. Like the parent comment, more bold statements about perceivable sound quality, with no evidence. How can it not be true as I am describing my experience. Are you really telling me that I actually can tell when I say I can't? It depends on the encoder, the track, your equipment, and how good you are at picking out artifacts.
Some people do surprisingly well in double-blind tests, though I doubt anyone can do it all the time on every sample. There is no scientific evidence that anyone can do it at all above kbps. Freaky on Dec 14, root parent next [—]. This is why ABX testing is so big in lossy audio circles. People can and do demonstrate their ability to distinguish between lossy and lossless encodings with certain samples in double-blind tests, at all sorts of bitrates.
I've done it myself occasionally. That people have been doing this for many years is one of the big reasons modern encoders are so good - they've needed tonnes of careful tuning to get to this point. You are making some bold statements about the general transparency of different audio formats that contradict pretty much everything I've read about this topic so far. Hence, I'd like to learn more, do you have any links that you would recommend?
Well, try it yourself :. Make sure to make it blind-test with the help of somebody else. Ideally such things would be subject to scientific studies. But these are kind of expensive and nobody cares for mp3 anyway. I'm not aware of any recent. Hydrogenaudio listening tests [1] are studies by volunteers, but they focus on non-transparent compression. Anyway, it also illustrate aswell how bad mp3 is. This page says at Kbps all the encoders were the same a 5 way tie. Their wiki says at Kbps MP3 is transparent [1].
You realize that that test doesn't test anywhere near kbps mp3, right? Yes, that's what the last two sentences are about. And it was reproducible on all my computers. I couldn't use my opus collection at all because either the encoder was broken or the playback was broken. I need to do it again at some point, when I have 8 hours to transcode everything and try again. See if they've fixed it. Don't ever do that. If you didn't rip CDs to lossless or buy lossless, you are stuck with the format you've got.
Using xz for Linux builds of your software might make sense though, or would do so in the future. So I'd say the majority of Linux users already have access to xz.
I just started typing in 'xz compression' in Google to learn more about it, it offered 'xz compression not available'. And some more queries that indicate it's not quite ubiquitous. It's good to be informed about the capabilities of xz.
I will keep using gzip, but consider xz in situations where the size or time matters. I might not care about megs versus 50 very much, but I will about two gigabytes versus one. Is it likely that a user has gzip on a system but not tar itself?
From the article: What about tooling? OSX: tar -xf some. Linux: tar -xf some. No idea, I haven't touched the platform in a while Tar does not implement decompression. If you don't have xz installed it won't work.
That doesn't seem correct, on osx AFAIK So tar does "implement decompression" and compression, by delegating the work to libarchive and it can work even "if you don't have xz installed". It would require liblzma, but you are correct that the library is a separate thing from the executable xz. IIRC bsdtar e.
On the vast majority of Linux distributions, you can pretty much guarantee that both tar and zlib will be installed. Both tend to be part of an essential core of packages required to install a system.
Pretty sure tar -xf does not actually work on osx unless you download a recent tar. M4v3R on Dec 12, root parent next [—]. I have tar that came with the system latest OSX and tar -xf works just fine.
And it did work fine for as long as I can remember. It does, and should have since at least I'd argue that bzip2 is a better example of a compression algorithm which no one needs anymore. Bzip2 isn't the best at anything anymore. You can easily apply the same argument to xz here, by introducing something rarer with an even better compression ratio e.
Now xz isn't the best at anything either. But despite zpaq being public domain, few people have heard of it and the debian package is ancient, and so the ubiquity argument really does count for something after all. No, xz on a particular level setting is both faster than bzip2 and provides better compression ratio, but zpaq is just slower.
Are you implying that xz out-compresses zpaq? Can you supply a benchmark? No, where did you find this implication in my comment? It seems like it can be faster and give better compression ratio than bzip2, can it? Two points: 1 It's very, very easy for the best solution to a problem not to simultaneously be the best along any single dimension.
If you see a spectrum where each dimension has a unique 1 and all the 2s are the same thing, that 2 solution is pretty likely to be the best of all the solutions. Your hypothetical example does actually make a compelling argument that bzip2 is useless, but that's not because it doesn't come in 1 anywhere; it's because it comes in behind xz everywhere.
Except ubiquity, but that's likely to change pretty quickly in the face of total obsolescence. But that something is compression and decompression speed, and if your only goal is to optimize those you can do much better by not using lzop 0 milliseconds to compress and decompress! So that's actually a terrible hypothetical result for lzop.
Heck, zero compression easily wins three of your four categories. Zero compression is very often the correct choice these days. No even when speed matters sometime lz4 is the best answer. I wrote a data sync that worked over mbps WAN and using lz4 on the serialised data transferred far faster than the raw data. Being second-best on ratio and ubiquity is still pretty handy for serving files. It's compress-once, decompress on somebody else's machine, so neither of those matter.
Ratio saves you money and ubiquity means people can actually use the file. The contents of that mount was used by the web application. Deletion was not an option. Something that compressed quickly was needed. And on the retrieval end, when the web app needs to do decompression, seconds matter. There are file splitters available for this. Well, it doesn't have to be split up, it's just that the reported sizes will be wrong. Peter Shannon Peter Shannon 21 1 1 bronze badge.
Sign up or log in Sign up using Google. Sign up using Facebook. Sign up using Email and Password. Post as a guest Name. Email Required, but never shown. The Overflow Blog. Podcast Explaining the semiconductor shortage, and how it might end. Does ES6 make JavaScript frameworks obsolete? Featured on Meta.
Now live: A fully responsive profile.
0コメント