w4a8 quants?

#7
by HAV0X1014 - opened

w4a8 is what I like to use on a 8gb vram GPU as it is smaller than int8 when stored but activates in int8 size. it is quite nice to have as an option for smaller GPUs without resorting to GGUF quantizations which can be slower than int8 in some cases.

Sign up or log in to comment