- 标题
- 摘要
- 关键词
- 实验方案
- 产品
-
[IEEE 2018 25th IEEE International Conference on Image Processing (ICIP) - Athens, Greece (2018.10.7-2018.10.10)] 2018 25th IEEE International Conference on Image Processing (ICIP) - Semantically Invariant Text-to-Image Generation
摘要： Image captioning has demonstrated models that are capable of generating plausible text given input images or videos. Further, recent work in image generation has shown significant improvements in image quality when text is used as a prior. Our work ties these concepts together by creating an architecture that can enable bidirectional generation of images and text. We call this network Multi-Modal Vector Representation (MMVR). Along with MMVR, we propose two improvements to the text conditioned image generation. Firstly, an n-gram metric based cost function is introduced that generalizes the caption with respect to the image. Secondly, multiple semantically similar sentences are shown to help in generating better images. Qualitative and quantitative evaluations demonstrate that MMVR improves upon existing text conditioned image generation results by over 20%, while integrating visual and text modalities.
关键词： text-to-image generation,multi-modal learning,n-gram metric,semantic invariance,multiple captions
更新于2025-09-23 15:22:29
-
[IEEE IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium - Valencia (2018.7.22-2018.7.27)] IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium - Deep Semantic Segmentation of Aerial Imagery Based on Multi-Modal Data
摘要： In this paper, we focus on the use of multi-modal data to achieve a semantic segmentation of aerial imagery. Thereby, the multi-modal data is composed of a true orthophoto, the Digital Surface Model (DSM) and further representations derived from these. Taking data of different modalities separately and in combination as input to a Residual Shuffling Convolutional Neural Network (RSCNN), we analyze their value for the classification task given with a benchmark dataset. The derived results reveal an improvement if different types of geometric features extracted from the DSM are used in addition to the true orthophoto.
关键词： multi-modal data,aerial imagery,Shuffling-CNN,deep learning,Semantic segmentation
更新于2025-09-23 15:22:29
-
Ultrasmall MoS2 Nanodots Doped Biodegradable SiO2 Nanoparticles for Clearable FL/CT/MSOT Imaging Guided PTT/PDT Combination Tumor therapy
摘要： Recently, we developed ultrasmall molybdenum disulfide (MoS2) quantum dots for computed tomography (CT) and multispectral optoacoustic tomography (MSOT) image-guided photothermal therapy (PTT). But, due to the rapid body elimination and the limited blood circulation time, the tumor uptake of the dots is low. In our study, this problem was solved via designing an amino modified biodegradable nanomaterial based on MoS2 quantum dots doped disulfide-based SiO2 nanoparticles (denoted MoS2@ss-SiO2) for multimodal application. By integrating the MoS2 quantum dots into clearable SiO2 nanoparticles, this nanoplatform with an appropriate particle size can not only degrade and excrete in a reasonable period induced by the redox-responsive with the glutathione (GSH), but also exhibits a high tumor uptake due to the longer blood circulation time. Moreover, hyaluronic acid (HA) and chlorin e6 (Ce6) were adsorbed on the outer shell for the tumor targeting effect and photodynamic therapy, respectively. So this biodegradable and clearable theranostic nanocomposite applicable in integrated fluorescence(FL)/CT/MSOT imaging guided combined photothermal therapy (PTT) and photodynamic therapy (PDT) is very optimistic in biomedical applications in the future.
关键词： biodegradable,multi-modal imaging,clearable,combination tumor therapy,ultrasmall MoS2 quantum dots
更新于2025-09-23 15:22:29
-
[IEEE 2018 IEEE SENSORS - New Delhi, India (2018.10.28-2018.10.31)] 2018 IEEE SENSORS - Smart Gas Sensing and Actuation Using Multimode of a MOFs Coated Microbeam
摘要： Smart sensing systems suffers complexity requiring interface circuits, microcontrollers, switches, and actuators to detect and sense, process the signal and take a decision, and trigger an action upon demand. This increases the device footprint and boosts significantly the power required to actuate the system. Here, we present a hybrid sensor and switch device, which is capable of accurately measuring gas concentration and perform switching when the concentration exceeds specific (safe) threshold. The device is based on a clamped-clamped microbeam coated with metal-organic frameworks (MOFs). Using the electrostatic harmonic voltage, we employ dynamic multi-modal actuation in which the microbeam is simultaneously excited at the first mode of vibration, near the pull-in band, and at the third mode. We demonstrate experimentally the effectiveness of this technique in measuring the concentration of water vapor and achieving switching when the concentration exceeds a threshold value. In contrast to a single mode operation, we show that employing multi-modal excitation improves sensitivity, enhances accuracy, and strengthen noise immunity.
关键词： Multi-modal,bifurcation,MOFs,Resonator,Smart sensor
更新于2025-09-19 17:15:36
-
X-ray Beam Induced Current Measurements for Multi-Modal X-ray Microscopy of Solar Cells
摘要： X-ray beam induced current (XBIC) measurements allow mapping of the nanoscale performance of electronic devices such as solar cells. Ideally, XBIC is employed simultaneously with other techniques within a multi-modal X-ray microscopy approach. An example is given herein combining XBIC with X-ray fluorescence to enable point-by-point correlations of the electrical performance with chemical composition. For the highest signal-to-noise ratio in XBIC measurements, lock-in amplification plays a crucial role. By this approach, the X-ray beam is modulated by an optical chopper upstream of the sample. The modulated X-ray beam induced electrical signal is amplified and demodulated to the chopper frequency using a lock-in amplifier. By optimizing low-pass filter settings, modulation frequency, and amplification amplitudes, noise can efficiently be suppressed for the extraction of a clear XBIC signal. A similar setup can be used to measure the X-ray beam induced voltage (XBIV). Beyond standard XBIC/XBIV measurements, XBIC can be measured with bias light or bias voltage applied such that outdoor working conditions of solar cells can be reproduced during in-situ and operando measurements. Ultimately, the multi-modal and multi-dimensional evaluation of electronic devices at the nanoscale enables new insights into the complex dependencies between composition, structure, and performance, which is an important step towards solving the materials' paradigm.
关键词： XBIV,solar cell,lock-in amplification,synchrotron radiation,photovoltaics,multi-modal,XBIC,CIGS,X-ray beam induced voltage,X-ray microscopy,X-ray beam induced current
更新于2025-09-16 10:30:52
-
Deep Coupled ISTA Network for Multi-modal Image Super-Resolution
摘要： Given a low-resolution (LR) image, multi-modal image super-resolution (MISR) aims to find the high-resolution (HR) version of this image with the guidance of an HR image from another modality. In this paper, we use a model-based approach to design a new deep network architecture for MISR. We first introduce a novel joint multi-modal dictionary learning (JMDL) algorithm to model cross-modality dependency. In JMDL, we simultaneously learn three dictionaries and two transform matrices to combine the modalities. Then, by unfolding the iterative shrinkage and thresholding algorithm (ISTA), we turn the JMDL model into a deep neural network, called deep coupled ISTA network. Since the network initialization plays an important role in deep network training, we further propose a layer-wise optimization algorithm (LOA) to initialize the parameters of the network before running back-propagation strategy. Specifically, we model the network initialization as a multi-layer dictionary learning problem, and solve it through convex optimization. The proposed LOA is demonstrated to effectively decrease the training loss and increase the reconstruction accuracy. Finally, we compare our method with other state-of-the-art methods in the MISR task. The numerical results show that our method consistently outperforms others both quantitatively and qualitatively at different upscaling factors for various multi-modal scenarios.
关键词： ISTA,multi-modal image super-resolution,dictionary learning,deep neural network
更新于2025-09-16 10:30:52
-
[IEEE 2019 18th International Conference on Optical Communications and Networks (ICOCN) - Huangshan, China (2019.8.5-2019.8.8)] 2019 18th International Conference on Optical Communications and Networks (ICOCN) - Demonstration of Image Processing Based on Reinforcement Learning in Multi-Modal Optical Transport Networks
摘要： In optical transport networks, orchestrating concurrent services may consume extra O/E ports for optical-electric conversion, and thus will introduce excessive cost. We first demonstrate an image processing based reinforcement-learning algorithm for concurrent service orchestration in multi-modal optical networks, and simulation results show it can improve cost-efficiency by saving provisioned O/E ports.
关键词： Optical transport networks,service orchestration,reinforcement learning,multi-modal
更新于2025-09-16 10:30:52
-
Defect activation and annihilation in CIGS solar cells: an operando X-ray microscopy study
摘要： The efficiency of thin-film solar cells with a Cu(In1?xGax)Se2 absorber is limited by nanoscopic inhomogeneities and defects. Traditional characterization methods are challenged by the multi-scale evaluation of the performance at defects that are buried in the device structures. Multi-modal X-ray microscopy offers a unique tool-set to probe the performance in fully assembled solar cells, and to correlate the performance with composition down to the micro- and nanoscale. We applied this approach to the mapping of temperature-dependent recombination for Cu(In1?xGax)Se2 solar cells with different absorber grain sizes, evaluating the same areas from room temperature to 100 ?C. It was found that poor performing areas in the large-grain sample are correlated with a Cu-deficient phase, whereas defects in the small-grain sample are not correlated with the distribution of Cu. In both samples, classes of recombination sites were identified, where defects were activated or annihilated by temperature. More generally, the methodology of combined operando and in-situ X-ray microscopy was established at the physical limit of spatial resolution given by the device itself. As proof-of-principle, the measurement of nanoscopic current generation in a solar cell is demonstrated with applied bias voltage and bias light.
关键词： X-ray beam induced current (XBIC),solar cell,Multi-modal X-ray microscopy,X-ray fluorescence (XRF),Cu(In1?xGax)Se2 (CIGS),X-ray beam induced voltage (XBIV)
更新于2025-09-12 10:27:22
-
Non-invasive imaging reveals conditions that impact distribution and persistence of cells after in vivo administration
摘要： Background: Cell-based regenerative medicine therapies are now frequently tested in clinical trials. In many conditions, cell therapies are administered systemically, but there is little understanding of their fate, and adverse events are often under-reported. Currently, it is only possible to assess safety and fate of cell therapies in preclinical studies, specifically by monitoring animals longitudinally using multi-modal imaging approaches. Here, using a suite of in vivo imaging modalities to explore the fate of a range of human and murine cells, we investigate how route of administration, cell type and host immune status affect the fate of administered cells. Methods: We applied a unique imaging platform combining bioluminescence, optoacoustic and magnetic resonance imaging modalities to assess the safety of different human and murine cell types by following their biodistribution and persistence in mice following administration into the venous or arterial system. Results: Longitudinal imaging analyses (i) suggested that the intra-arterial route may be more hazardous than intravenous administration for certain cell types, (ii) revealed that the potential of a mouse mesenchymal stem/stromal cell (MSC) line to form tumours depended on administration route and mouse strain and (iii) indicated that clinically tested human umbilical cord (hUC)-derived MSCs can transiently and unexpectedly proliferate when administered intravenously to mice. Conclusions: In order to perform an adequate safety assessment of potential cell-based therapies, a thorough understanding of cell biodistribution and fate post administration is required. The non-invasive imaging platform used here can expose not only the general organ distribution of these therapies, but also a detailed view of their presence within different organs and, importantly, tumourigenic potential. Our observation that the hUC-MSCs but not the human bone marrow (hBM)-derived MSCs persisted for a period in some animals suggests that therapies with these cells should proceed with caution.
关键词： Safety,Mesenchymal stem/stromal cells,Preclinical models,Multi-modal imaging,Cell therapies,Cell tracking
更新于2025-09-11 14:15:04
-
[IEEE 2018 10th IAPR Workshop on Pattern Recognition in Remote Sensing (PRRS) - Beijing (2018.8.19-2018.8.20)] 2018 10th IAPR Workshop on Pattern Recognition in Remote Sensing (PRRS) - Multi-Modal Remote Sensing Image Registration Based on Multi-Scale Phase Congruency
摘要： Automatic matching of multi-modal remote sensing images remains a challenging task in remote sensing image analysis due to significant non-linear radiometric differences between the phase congruency model with illumination and contrast invariance for image matching, and extends the model to a novel image registration method, named as multi-scale phase consistency (MS-PC). The Euclidean distance between MS-PC descriptors is used as similarity metric to achieve correspondences. The proposed method is evaluated with four pairs of multi-model remote sensing images. The experimental results show that MS-PC is more robust to the radiation differences between images, and performs better than two popular method (i.e. SIFT and SAR-SIFT) in both registration accuracy and tie points number.
关键词： Phase Congruency,Image Registration,Multi-modal Remote Sensing Image,Multi-scale
更新于2025-09-10 09:29:36