litert-community/M-LSD-tiny-LiteRT

M-LSD-tiny — LiteRT (on-device line segment detection, fully-GPU)

M-LSD (NAVER, AAAI 2022) light-weight real-time line segment detection, converted to LiteRT and running fully on the CompiledModel GPU (ML Drift) on Android. Detects straight line segments — building edges, document borders, wireframes, room layout. The tiny variant (MobileNetV2 backbone, 0.62M params) is 1.4 MB in fp16.

On-device (Pixel 8a, Tensor G3 — verified)

nodes on GPU	99 / 99 LITERT_CL (full residency)
inference	~2 ms (512×512)
size	1.4 MB (fp16)
accuracy	device-vs-PyTorch corr 0.997 (127 vs 128 lines decoded)

image[1,4,512,512] (RGB + ones channel, scaled to [-1,1]) →[GPU: MobileNetV2 U-Net]→ tpMap[1,9,256,256]

The output is a "TP map": channel 0 = line-center heatmap, channels 1–4 = start/end displacement. The decode (sigmoid + 3×3 NMS over centers, displacement → endpoints, ×2) runs on the host.

How it converts (litert-torch)

Pure CNN encoder-decoder. A single re-authoring: the decoder's F.interpolate(bilinear, align_corners=True) → align_corners=False (the Mali delegate bans align_corners=True + half-pixel). MobileNetV2 has no max-pool (strided convs → no PADV2), and the upsample is RESIZE_BILINEAR, not a transposed conv → fully GPU-clean. Result: banned ops NONE, all tensors ≤4D, tflite-vs-torch corr 1.0, device-vs-torch corr 0.997.

Preprocessing & decode

Resize to 512×512, append a 4th channel of ones, scale (x/127.5) - 1, NCHW. Decode: sigmoid the center map, 3×3 max NMS, threshold (0.10), displacement → endpoints, filter by length, ×2 to 512-space.

Minimal usage

Android (Kotlin, CompiledModel GPU)

val model = CompiledModel.create(context.assets, "mlsd_fp16.tflite",
    CompiledModel.Options(Accelerator.GPU), null)
val inputs = model.createInputBuffers(); val outputs = model.createOutputBuffers()
inputs[0].writeFloat(x)             // [1,4,512,512] NCHW: RGB + ones channel, x/127.5 - 1
model.run(inputs, outputs)
val tpMap = outputs[0].readFloat()  // [1,9,256,256]: ch0 center, ch1-4 displacement
// sigmoid + 3x3 NMS + displacement -> segments: port of the Python decode below.

Python (desktop verification)

import numpy as np
from PIL import Image
from scipy.ndimage import maximum_filter
from ai_edge_litert.interpreter import Interpreter

im = Image.open("photo.jpg").convert("RGB").resize((512, 512))
a = np.asarray(im, np.float32)
a = np.concatenate([a, np.ones((512, 512, 1), np.float32)], -1)   # 4th channel of ones
x = ((a.transpose(2, 0, 1)[None] / 127.5) - 1.0).copy()           # [1,4,512,512]

it = Interpreter(model_path="mlsd_fp16.tflite"); it.allocate_tensors()
it.set_tensor(it.get_input_details()[0]["index"], x); it.invoke()
tp = it.get_tensor(it.get_output_details()[0]["index"])[0]        # [9,256,256]

center = 1 / (1 + np.exp(-tp[0])); disp = tp[1:5]
peak = (center == maximum_filter(center, 3)) & (center > 0.10)    # 3x3 NMS + threshold
ys, xs = np.where(peak)
order = center[ys, xs].argsort()[::-1][:200]                      # top-200 centers
lines = []
for y, x0 in zip(ys[order], xs[order]):
    dxs, dys, dxe, dye = disp[:, y, x0]
    if np.hypot(dxs - dxe, dys - dye) > 20:                       # min segment length (px)
        lines.append([(x0 + dxs) * 2, (y + dys) * 2, (x0 + dxe) * 2, (y + dye) * 2])
print(f"{len(lines)} line segments (x0,y0,x1,y1 in 512-space)")

Performance

Measured on a Pixel 8a (Tensor G3, Android 16) with the standard TFLite benchmark_model tool — 10 warm-up runs then 50 timed runs, reported as the tool's mean.

Runtime	Backend	Graph on GPU	Latency
LiteRT CompiledModel (LITERT_CL)	GPU	99 / 99	~2 ms
TFLite benchmark_model (TfLiteGpuDelegateV2)	GPU (OpenCL)	99 / 99	26.3 ms
TFLite benchmark_model	CPU (XNNPACK, 4 threads)	—	XNNPACK declined the graph

The two GPU rows are different runtimes, not a contradiction. The LITERT_CL figure is the one recorded when this model shipped, taken through LiteRT's own CompiledModel accelerator — the path the Kotlin sample app and the LiteRT API use. The TfLiteGpuDelegateV2 figure is the classic TFLite OpenCL delegate, measured with a tool anyone can download and re-run. They agree on how much of the graph the GPU takes; they disagree on speed, and the classic delegate is the slower of the two here. Read the TfLiteGpuDelegateV2 row as a reproducible floor, not as this model's speed on LiteRT.

XNNPACK declines these fp16 graphs — it reports failed to delegate DEPTHWISE_CONV_2D and then fails to allocate tensors — so there is no usable CPU number. Disabling XNNPACK falls back to reference kernels, which measured about 20× slower than the GPU on models of this size and would not represent CPU inference anyone would ship.

License

Apache-2.0. Upstream: navervision/mlsd; PyTorch port lhwcv/mlsd_pytorch.